2024-03-26T06:13:05.184 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T06:13:05.184 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_rishabh@teuthology --archive /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} --verbose --name rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi -- /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752/orig.config.yaml 2024-03-26T06:13:05.242 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T06:13:05.329 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752 branch: rishabh-2024mar8-2 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} email: null first_in_suite: false job_id: '7622752' kernel: client: sha1: distro flavor: default kdb: true sha1: 1df0c7e8a2354c42d738e5454ba83ea7332deac2 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-2024mar8-2 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - SLOW_OPS - slow request sha1: 7e98f5f6b535defc7fa715b456904ffc0500aa9f subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 7e98f5f6b535defc7fa715b456904ffc0500aa9f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-2024mar8-2 sha1: 7e98f5f6b535defc7fa715b456904ffc0500aa9f owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1249 sha1: 7e98f5f6b535defc7fa715b456904ffc0500aa9f sleep_before_teardown: 0 subset: 481/512 suite: fs suite_branch: rishabh-2024mar8-2 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_7e98f5f6b535defc7fa715b456904ffc0500aa9f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 7e98f5f6b535defc7fa715b456904ffc0500aa9f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDK8IYk/l2NfYVn+XIYVJFZ/6zch67zvao5F9ftyWAOFb4QwIbBKQFDi7jvIcLbmXKXlvVeBZu4ywor6dCJ5w50= smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwYVQtiY5yFelKQqCSIngBczcLbFkGF67vYFFUCVmgEC3zPd8pah10Nytdx1k1EFMacI+/uvJgqrL1GXD9xyxo= smithi170.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH7jjFQL494AaApyg+lSA8QOSmia4/wk6Zer32WH9/pk8u9LEX75LQUUx2pNXlrn17IewuwLjGYpr9f54R6WRa4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:47:39 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T06:13:05.329 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_7e98f5f6b535defc7fa715b456904ffc0500aa9f/qa; will attempt to use it 2024-03-26T06:13:05.330 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_7e98f5f6b535defc7fa715b456904ffc0500aa9f/qa/tasks 2024-03-26T06:13:05.330 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T06:13:05.332 INFO:teuthology.task.internal:Checking packages... 2024-03-26T06:13:05.356 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7e98f5f6b535defc7fa715b456904ffc0500aa9f' 2024-03-26T06:13:05.356 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T06:13:05.356 INFO:teuthology.packaging:ref: None 2024-03-26T06:13:05.356 INFO:teuthology.packaging:tag: None 2024-03-26T06:13:05.357 INFO:teuthology.packaging:branch: rishabh-2024mar8-2 2024-03-26T06:13:05.357 INFO:teuthology.packaging:sha1: 7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:13:05.357 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-2024mar8-2 2024-03-26T06:13:05.502 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1964.g7e98f5f6 2024-03-26T06:13:05.504 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T06:13:05.510 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T06:13:05.510 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T06:13:05.550 INFO:teuthology.task.internal:Saving configuration 2024-03-26T06:13:05.570 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T06:13:05.576 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T06:13:05.603 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 06:08:11.920262', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDK8IYk/l2NfYVn+XIYVJFZ/6zch67zvao5F9ftyWAOFb4QwIbBKQFDi7jvIcLbmXKXlvVeBZu4ywor6dCJ5w50='} 2024-03-26T06:13:05.626 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 06:08:11.921872', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwYVQtiY5yFelKQqCSIngBczcLbFkGF67vYFFUCVmgEC3zPd8pah10Nytdx1k1EFMacI+/uvJgqrL1GXD9xyxo='} 2024-03-26T06:13:05.645 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 06:08:11.923122', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH7jjFQL494AaApyg+lSA8QOSmia4/wk6Zer32WH9/pk8u9LEX75LQUUx2pNXlrn17IewuwLjGYpr9f54R6WRa4='} 2024-03-26T06:13:05.645 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T06:13:05.652 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-26T06:13:05.652 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-26T06:13:05.653 INFO:teuthology.task.internal:roles: ubuntu@smithi170.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-26T06:13:05.653 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T06:13:05.764 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcf02c950d0>, signals=[15]) 2024-03-26T06:13:05.765 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T06:13:05.771 INFO:teuthology.task.internal:Opening connections... 2024-03-26T06:13:05.771 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T06:13:05.773 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T06:13:05.875 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-03-26T06:13:05.876 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T06:13:05.963 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi170.front.sepia.ceph.com 2024-03-26T06:13:05.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T06:13:06.048 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T06:13:06.056 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2024-03-26T06:13:06.074 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2024-03-26T06:13:06.074 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="CentOS Stream" 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="9" 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:ID="centos" 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="rhel fedora" 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="9" 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el9" 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:LOGO="fedora-logo-icon" 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://centos.org/" 2024-03-26T06:13:06.132 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T06:13:06.133 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T06:13:06.133 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T06:13:06.133 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2024-03-26T06:13:06.156 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-03-26T06:13:06.173 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-03-26T06:13:06.173 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-03-26T06:13:06.230 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-03-26T06:13:06.230 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="9" 2024-03-26T06:13:06.230 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-03-26T06:13:06.230 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-03-26T06:13:06.230 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="9" 2024-03-26T06:13:06.231 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el9" 2024-03-26T06:13:06.231 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T06:13:06.231 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-03-26T06:13:06.231 INFO:teuthology.orchestra.run.smithi121.stdout:LOGO="fedora-logo-icon" 2024-03-26T06:13:06.231 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T06:13:06.231 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-03-26T06:13:06.231 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T06:13:06.231 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T06:13:06.231 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T06:13:06.231 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-03-26T06:13:06.256 DEBUG:teuthology.orchestra.run.smithi170:> uname -m 2024-03-26T06:13:06.273 INFO:teuthology.orchestra.run.smithi170.stdout:x86_64 2024-03-26T06:13:06.274 DEBUG:teuthology.orchestra.run.smithi170:> cat /etc/os-release 2024-03-26T06:13:06.329 INFO:teuthology.orchestra.run.smithi170.stdout:NAME="CentOS Stream" 2024-03-26T06:13:06.329 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION="9" 2024-03-26T06:13:06.329 INFO:teuthology.orchestra.run.smithi170.stdout:ID="centos" 2024-03-26T06:13:06.329 INFO:teuthology.orchestra.run.smithi170.stdout:ID_LIKE="rhel fedora" 2024-03-26T06:13:06.329 INFO:teuthology.orchestra.run.smithi170.stdout:VERSION_ID="9" 2024-03-26T06:13:06.329 INFO:teuthology.orchestra.run.smithi170.stdout:PLATFORM_ID="platform:el9" 2024-03-26T06:13:06.329 INFO:teuthology.orchestra.run.smithi170.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T06:13:06.329 INFO:teuthology.orchestra.run.smithi170.stdout:ANSI_COLOR="0;31" 2024-03-26T06:13:06.329 INFO:teuthology.orchestra.run.smithi170.stdout:LOGO="fedora-logo-icon" 2024-03-26T06:13:06.329 INFO:teuthology.orchestra.run.smithi170.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T06:13:06.329 INFO:teuthology.orchestra.run.smithi170.stdout:HOME_URL="https://centos.org/" 2024-03-26T06:13:06.330 INFO:teuthology.orchestra.run.smithi170.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T06:13:06.330 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T06:13:06.330 INFO:teuthology.orchestra.run.smithi170.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T06:13:06.330 INFO:teuthology.lock.ops:Updating smithi170.front.sepia.ceph.com on lock server 2024-03-26T06:13:06.347 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T06:13:06.357 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T06:13:06.375 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T06:13:06.376 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2024-03-26T06:13:06.378 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-03-26T06:13:06.381 DEBUG:teuthology.orchestra.run.smithi170:> test '!' -e /home/ubuntu/cephtest 2024-03-26T06:13:06.397 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T06:13:06.404 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T06:13:06.404 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2024-03-26T06:13:06.435 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-03-26T06:13:06.438 DEBUG:teuthology.orchestra.run.smithi170:> test -z $(ls -A /var/lib/ceph) 2024-03-26T06:13:06.552 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T06:13:06.622 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T06:13:06.636 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '1df0c7e8a2354c42d738e5454ba83ea7332deac2'} 2024-03-26T06:13:06.636 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-26T06:13:06.636 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-26T06:13:06.637 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T06:13:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:13:06.654 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-03-26T06:13:06.712 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-361.el9.x86_64 2024-03-26T06:13:06.712 INFO:teuthology.task.kernel:Running kernel on smithi038: 5.14.0-361.el9.x86_64 2024-03-26T06:13:06.712 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2024-03-26T06:13:08.718 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-26T06:13:10.143 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-26T06:13:10.926 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-26T06:13:13.594 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-26T06:13:17.870 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-26T06:13:19.287 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 16 kB 00:00 2024-03-26T06:13:21.439 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-26T06:13:26.597 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-26T06:13:28.804 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T06:13:28.805 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T06:13:28.889 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-26T06:13:28.895 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-26T06:13:28.895 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2024-03-26T06:13:28.895 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-26T06:13:28.895 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-26T06:13:28.895 INFO:teuthology.orchestra.run.smithi038.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T06:13:28.895 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-26T06:13:28.895 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T06:13:28.895 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T06:13:28.895 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T06:13:28.895 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:13:28.896 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-26T06:13:28.896 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2024-03-26T06:13:28.896 INFO:teuthology.orchestra.run.smithi038.stdout:Install 4 Packages 2024-03-26T06:13:28.896 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:13:28.896 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 76 M 2024-03-26T06:13:28.896 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 123 M 2024-03-26T06:13:28.896 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-26T06:13:29.415 INFO:teuthology.orchestra.run.smithi038.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 166 kB/s | 66 kB 00:00 2024-03-26T06:13:31.324 INFO:teuthology.orchestra.run.smithi038.stdout:(2/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 15 MB/s | 33 MB 00:02 2024-03-26T06:13:41.673 INFO:teuthology.orchestra.run.smithi038.stdout:(3/4): kernel-core-5.14.0-432.el9.x86_64.rpm 1.2 MB/s | 15 MB 00:12 2024-03-26T06:13:51.923 INFO:teuthology.orchestra.run.smithi038.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 1.2 MB/s | 27 MB 00:22 2024-03-26T06:13:51.923 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-26T06:13:51.923 INFO:teuthology.orchestra.run.smithi038.stdout:Total 3.3 MB/s | 76 MB 00:23 2024-03-26T06:13:52.619 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-26T06:13:53.060 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-26T06:13:53.061 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-26T06:13:54.105 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-26T06:13:54.105 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-26T06:13:56.717 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-26T06:13:57.342 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T06:13:57.449 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T06:14:00.364 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T06:14:00.514 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T06:14:05.590 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T06:14:05.631 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T06:14:10.750 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T06:14:41.832 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T06:14:41.832 INFO:teuthology.orchestra.run.smithi038.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T06:14:41.832 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:14:41.878 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T06:14:42.211 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T06:14:42.212 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T06:14:42.212 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T06:14:42.212 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T06:14:42.671 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T06:14:42.671 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:14:42.671 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-26T06:14:42.671 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T06:14:42.671 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T06:14:42.671 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T06:14:42.671 INFO:teuthology.orchestra.run.smithi038.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T06:14:42.671 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:14:42.671 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-26T06:14:42.993 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T06:14:43.113 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T06:14:43.114 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T06:14:43.114 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T06:14:43.114 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T06:14:43.114 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T06:14:43.114 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-26T06:14:43.114 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi038.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T06:14:43.115 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T06:14:43.197 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T06:14:43.197 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T06:14:43.198 DEBUG:teuthology.orchestra.run.smithi038:> sudo rpm -qi grub2-tools 2024-03-26T06:14:43.243 INFO:teuthology.orchestra.run.smithi038.stdout:Name : grub2-tools 2024-03-26T06:14:43.243 INFO:teuthology.orchestra.run.smithi038.stdout:Epoch : 1 2024-03-26T06:14:43.243 INFO:teuthology.orchestra.run.smithi038.stdout:Version : 2.06 2024-03-26T06:14:43.243 INFO:teuthology.orchestra.run.smithi038.stdout:Release : 68.el9 2024-03-26T06:14:43.243 INFO:teuthology.orchestra.run.smithi038.stdout:Architecture: x86_64 2024-03-26T06:14:43.243 INFO:teuthology.orchestra.run.smithi038.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T06:14:43.243 INFO:teuthology.orchestra.run.smithi038.stdout:Group : Unspecified 2024-03-26T06:14:43.244 INFO:teuthology.orchestra.run.smithi038.stdout:Size : 8264254 2024-03-26T06:14:43.244 INFO:teuthology.orchestra.run.smithi038.stdout:License : GPLv3+ 2024-03-26T06:14:43.244 INFO:teuthology.orchestra.run.smithi038.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T06:14:43.244 INFO:teuthology.orchestra.run.smithi038.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T06:14:43.244 INFO:teuthology.orchestra.run.smithi038.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T06:14:43.244 INFO:teuthology.orchestra.run.smithi038.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T06:14:43.244 INFO:teuthology.orchestra.run.smithi038.stdout:Packager : builder@centos.org 2024-03-26T06:14:43.244 INFO:teuthology.orchestra.run.smithi038.stdout:Vendor : CentOS 2024-03-26T06:14:43.244 INFO:teuthology.orchestra.run.smithi038.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T06:14:43.245 INFO:teuthology.orchestra.run.smithi038.stdout:Summary : Support tools for GRUB. 2024-03-26T06:14:43.245 INFO:teuthology.orchestra.run.smithi038.stdout:Description : 2024-03-26T06:14:43.245 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:14:43.245 INFO:teuthology.orchestra.run.smithi038.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T06:14:43.245 INFO:teuthology.orchestra.run.smithi038.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T06:14:43.245 INFO:teuthology.orchestra.run.smithi038.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T06:14:43.245 INFO:teuthology.orchestra.run.smithi038.stdout:hardware devices. 2024-03-26T06:14:43.245 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:14:43.245 INFO:teuthology.orchestra.run.smithi038.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T06:14:43.247 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T06:14:43.247 INFO:teuthology.task.kernel:Updating grub on smithi038 to boot 5.14.0-432.el9.x86_64 2024-03-26T06:14:43.247 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T06:14:43.863 INFO:teuthology.orchestra.run.smithi038.stderr:Generating grub configuration file ... 2024-03-26T06:14:45.298 INFO:teuthology.orchestra.run.smithi038.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T06:14:45.332 INFO:teuthology.orchestra.run.smithi038.stderr:done 2024-03-26T06:14:45.344 DEBUG:teuthology.orchestra.run.smithi038:> mktemp 2024-03-26T06:14:45.399 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/tmp.VBZUJm2nuC 2024-03-26T06:14:45.400 DEBUG:teuthology.orchestra.run.smithi038:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VBZUJm2nuC 2024-03-26T06:14:45.479 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0666 /tmp/tmp.VBZUJm2nuC 2024-03-26T06:14:45.620 DEBUG:teuthology.orchestra.remote:smithi038:/tmp/tmp.VBZUJm2nuC is 6KB 2024-03-26T06:14:45.668 DEBUG:teuthology.orchestra.run.smithi038:> rm -fr /tmp/tmp.VBZUJm2nuC 2024-03-26T06:14:45.683 DEBUG:teuthology.orchestra.run.smithi038:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T06:14:45.751 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T06:14:45.751 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T06:14:45.752 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T06:14:45.752 INFO:teuthology.orchestra.run.smithi038.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T06:14:45.753 DEBUG:teuthology.orchestra.run.smithi038:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T06:14:45.891 DEBUG:teuthology.orchestra.run.smithi038:> sudo shutdown -r now 2024-03-26T06:15:15.893 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-26T06:15:15.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T06:15:15.895 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T06:15:34.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2024-03-26T06:15:43.431 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T06:15:43.432 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T06:15:46.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.38 2024-03-26T06:15:58.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T06:15:58.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T06:16:30.388 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-03-26T06:16:30.741 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T06:16:30.741 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-432.el9.x86_64"... 2024-03-26T06:16:30.741 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2024-03-26T06:16:30.756 INFO:teuthology.orchestra.run.smithi038.stdout:5.14.0-432.el9.x86_64 2024-03-26T06:16:30.756 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T06:16:30.756 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T06:16:30.756 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T06:16:31.758 DEBUG:teuthology.parallel:result is None 2024-03-26T06:16:31.758 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T06:16:31.765 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T06:16:31.766 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T06:16:31.768 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T06:16:31.770 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T06:16:31.788 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T06:16:31.801 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T06:16:31.835 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T06:16:31.835 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T06:16:31.838 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T06:16:31.840 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T06:16:31.956 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T06:16:31.963 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T06:16:31.963 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T06:16:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:16:31.980 DEBUG:teuthology.orchestra.run.smithi170:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T06:16:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:16:31.997 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T06:16:31.999 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T06:16:32.023 DEBUG:teuthology.orchestra.run.smithi170:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T06:16:32.082 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T06:16:32.087 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T06:16:32.102 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T06:16:32.108 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T06:16:32.127 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T06:16:32.154 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T06:16:32.157 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T06:16:32.164 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T06:16:32.164 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T06:16:32.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T06:16:32.169 DEBUG:teuthology.orchestra.run.smithi170:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T06:16:32.199 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T06:16:32.208 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T06:16:32.209 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T06:16:32.239 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T06:16:32.242 DEBUG:teuthology.orchestra.run.smithi170:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T06:16:32.258 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T06:16:32.338 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T06:16:32.384 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T06:16:32.470 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T06:16:32.514 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:16:32.514 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T06:16:32.584 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T06:16:32.627 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T06:16:32.680 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T06:16:32.764 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T06:16:32.808 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-26T06:16:32.809 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T06:16:32.877 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T06:16:32.924 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T06:16:32.982 DEBUG:teuthology.orchestra.run.smithi170:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T06:16:33.067 DEBUG:teuthology.orchestra.run.smithi170:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T06:16:33.111 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-26T06:16:33.111 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T06:16:33.180 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2024-03-26T06:16:33.183 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-03-26T06:16:33.186 DEBUG:teuthology.orchestra.run.smithi170:> sudo service rsyslog restart 2024-03-26T06:16:33.224 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T06:16:33.236 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T06:16:33.263 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T06:16:33.694 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T06:16:33.727 INFO:teuthology.task.internal:Starting timer... 2024-03-26T06:16:33.728 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T06:16:33.776 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T06:16:33.798 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-26T06:16:33.798 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2024-03-26T06:16:33.852 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2024-03-26T06:16:33.854 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-03-26T06:16:33.917 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-03-26T06:16:33.919 DEBUG:teuthology.orchestra.run.smithi170:> sudo service auditd rotate 2024-03-26T06:16:33.987 INFO:teuthology.orchestra.run.smithi170.stdout:Rotating logs: 2024-03-26T06:16:33.989 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T06:16:33.989 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2024-03-26T06:16:34.011 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2024-03-26T06:16:34.012 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-03-26T06:16:34.039 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-03-26T06:16:34.040 DEBUG:teuthology.orchestra.run.smithi170:> /usr/sbin/getenforce 2024-03-26T06:16:34.064 INFO:teuthology.orchestra.run.smithi170.stdout:Permissive 2024-03-26T06:16:34.064 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi170.front.sepia.ceph.com': 'permissive'} 2024-03-26T06:16:34.064 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T06:16:34.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:16:34.093 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T06:16:34.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:16:34.123 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T06:16:34.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:16:34.152 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T06:16:34.152 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2024-03-26T06:16:34.188 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-03-26T06:16:34.222 DEBUG:teuthology.orchestra.run.smithi170:> sudo /usr/sbin/setenforce permissive 2024-03-26T06:16:34.266 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T06:16:34.276 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-26T06:16:34.281 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-26T06:16:34.418 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T06:16:34.436 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T06:16:34.438 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi121.front.sepia.ceph.com,smithi170.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T06:24:11.930 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi170.front.sepia.ceph.com')] 2024-03-26T06:24:11.951 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T06:24:11.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T06:24:12.042 DEBUG:teuthology.orchestra.run.smithi038:> true 2024-03-26T06:24:12.120 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi038.front.sepia.ceph.com' 2024-03-26T06:24:12.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-26T06:24:12.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T06:24:12.210 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-03-26T06:24:12.281 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-03-26T06:24:12.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-26T06:24:12.282 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi170.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T06:24:12.369 DEBUG:teuthology.orchestra.run.smithi170:> true 2024-03-26T06:24:12.438 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi170.front.sepia.ceph.com' 2024-03-26T06:24:12.438 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T06:24:12.451 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T06:24:12.451 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T06:24:12.452 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T06:24:12.454 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T06:24:12.454 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T06:24:12.456 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T06:24:12.457 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T06:24:12.486 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T06:24:12.487 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T06:24:12.504 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T06:24:12.504 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T06:24:12.513 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T06:24:12.530 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T06:24:12.533 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-03-26T06:24:12.534 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2024-03-26T06:24:12.547 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-03-26T06:24:12.549 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2024-03-26T06:24:12.560 INFO:teuthology.orchestra.run.smithi170.stderr:sudo: ntpd: command not found 2024-03-26T06:24:12.563 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T06:24:12.566 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T06:24:12.574 INFO:teuthology.orchestra.run.smithi170.stdout:506 Cannot talk to daemon 2024-03-26T06:24:12.579 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T06:24:12.582 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T06:24:12.591 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T06:24:12.608 INFO:teuthology.orchestra.run.smithi170.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T06:24:12.623 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-03-26T06:24:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T06:24:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-03-26T06:24:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.626 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.632 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-03-26T06:24:12.635 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T06:24:12.635 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-03-26T06:24:12.635 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.635 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.635 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.636 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.652 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-26T06:24:12.654 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T06:24:12.655 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-26T06:24:12.655 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.655 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.655 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.655 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:24:12.656 INFO:teuthology.run_tasks:Running task install... 2024-03-26T06:24:12.665 DEBUG:teuthology.task.install:project ceph 2024-03-26T06:24:12.665 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7e98f5f6b535defc7fa715b456904ffc0500aa9f'}} 2024-03-26T06:24:12.665 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '7e98f5f6b535defc7fa715b456904ffc0500aa9f'} 2024-03-26T06:24:12.665 INFO:teuthology.task.install:Using flavor: default 2024-03-26T06:24:12.671 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T06:24:12.671 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-26T06:24:12.672 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7e98f5f6b535defc7fa715b456904ffc0500aa9f', 'tag': None, 'wait_for_package': False} 2024-03-26T06:24:12.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:24:12.678 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7e98f5f6b535defc7fa715b456904ffc0500aa9f', 'tag': None, 'wait_for_package': False} 2024-03-26T06:24:12.678 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:24:12.679 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7e98f5f6b535defc7fa715b456904ffc0500aa9f', 'tag': None, 'wait_for_package': False} 2024-03-26T06:24:12.679 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:24:12.822 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/ 2024-03-26T06:24:12.822 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1964.g7e98f5f6 2024-03-26T06:24:12.834 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/ 2024-03-26T06:24:12.834 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1964.g7e98f5f6 2024-03-26T06:24:12.842 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/ 2024-03-26T06:24:12.842 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1964.g7e98f5f6 2024-03-26T06:24:13.038 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T06:24:13.039 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:24:13.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T06:24:13.052 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T06:24:13.052 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-26T06:24:13.052 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T06:24:13.060 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T06:24:13.060 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-26T06:24:13.060 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T06:24:13.068 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-26T06:24:13.069 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7e98f5f6b535defc7fa715b456904ffc0500aa9f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T06:24:13.080 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-26T06:24:13.080 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7e98f5f6b535defc7fa715b456904ffc0500aa9f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T06:24:13.090 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-26T06:24:13.090 DEBUG:teuthology.orchestra.run.smithi170:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7e98f5f6b535defc7fa715b456904ffc0500aa9f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T06:24:13.139 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T06:24:13.148 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T06:24:13.158 DEBUG:teuthology.orchestra.run.smithi170:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T06:24:13.221 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T06:24:13.227 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T06:24:13.237 DEBUG:teuthology.orchestra.run.smithi170:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T06:24:13.290 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2024-03-26T06:24:13.290 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2024-03-26T06:24:13.293 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-03-26T06:24:13.295 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-03-26T06:24:13.303 INFO:teuthology.orchestra.run.smithi170.stdout:check_obsoletes = 1 2024-03-26T06:24:13.305 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum clean all 2024-03-26T06:24:13.528 INFO:teuthology.orchestra.run.smithi121.stdout:74 files removed 2024-03-26T06:24:13.541 INFO:teuthology.orchestra.run.smithi170.stdout:74 files removed 2024-03-26T06:24:13.541 INFO:teuthology.orchestra.run.smithi038.stdout:74 files removed 2024-03-26T06:24:13.555 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-26T06:24:13.567 DEBUG:teuthology.orchestra.run.smithi170:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-26T06:24:13.568 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-26T06:24:14.155 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 264 kB/s | 82 kB 00:00 2024-03-26T06:24:14.257 INFO:teuthology.orchestra.run.smithi170.stdout:ceph packages for x86_64 208 kB/s | 82 kB 00:00 2024-03-26T06:24:14.356 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 170 kB/s | 82 kB 00:00 2024-03-26T06:24:14.506 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-26T06:24:14.665 INFO:teuthology.orchestra.run.smithi170.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-26T06:24:14.748 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-26T06:24:14.940 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-03-26T06:24:14.982 INFO:teuthology.orchestra.run.smithi170.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-03-26T06:24:15.182 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 4.8 kB/s | 1.9 kB 00:00 2024-03-26T06:24:15.367 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-26T06:24:15.485 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-26T06:24:15.642 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-26T06:24:16.976 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-26T06:24:17.102 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-26T06:24:17.193 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-26T06:24:17.933 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-26T06:24:17.959 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-26T06:24:19.593 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-03-26T06:24:22.985 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - AppStream 4.9 MB/s | 19 MB 00:03 2024-03-26T06:24:24.901 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-26T06:24:26.360 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-26T06:24:28.452 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-26T06:24:28.514 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-26T06:24:30.929 INFO:teuthology.orchestra.run.smithi038.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-26T06:24:33.711 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 5.2 kB/s | 1.7 kB 00:00 2024-03-26T06:24:35.213 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 5.0 MB/s | 21 MB 00:04 2024-03-26T06:24:35.290 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - BaseOS 461 kB/s | 8.0 MB 00:17 2024-03-26T06:24:36.186 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-26T06:24:36.187 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-26T06:24:36.201 INFO:teuthology.orchestra.run.smithi121.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-26T06:24:36.202 INFO:teuthology.orchestra.run.smithi121.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-26T06:24:36.203 INFO:teuthology.orchestra.run.smithi121.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-26T06:24:36.204 INFO:teuthology.orchestra.run.smithi121.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-26T06:24:36.206 INFO:teuthology.orchestra.run.smithi121.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-26T06:24:36.209 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-26T06:24:36.210 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-26T06:24:36.288 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-03-26T06:24:36.300 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-26T06:24:36.300 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-03-26T06:24:36.300 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 6.6 k 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 5.4 M 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 889 k 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 145 k 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 1.5 M 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 139 k 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 1.8 M 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 7.4 M 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 49 k 2024-03-26T06:24:36.301 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 9.8 M 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 47 M 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 264 k 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 749 k 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 219 k 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-shell x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 27 k 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 18 k 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 31 k 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 745 k 2024-03-26T06:24:36.302 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 127 k 2024-03-26T06:24:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-26T06:24:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-26T06:24:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-26T06:24:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-26T06:24:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-26T06:24:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-26T06:24:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 162 k 2024-03-26T06:24:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 321 k 2024-03-26T06:24:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 298 k 2024-03-26T06:24:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 99 k 2024-03-26T06:24:36.303 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 86 k 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 3.1 M 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 170 k 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-26T06:24:36.304 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-26T06:24:36.305 INFO:teuthology.orchestra.run.smithi121.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-26T06:24:36.305 INFO:teuthology.orchestra.run.smithi121.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-26T06:24:36.305 INFO:teuthology.orchestra.run.smithi121.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-26T06:24:36.305 INFO:teuthology.orchestra.run.smithi121.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-26T06:24:36.305 INFO:teuthology.orchestra.run.smithi121.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-26T06:24:36.305 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 3.4 M 2024-03-26T06:24:36.305 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 3.1 M 2024-03-26T06:24:36.305 INFO:teuthology.orchestra.run.smithi121.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-26T06:24:36.305 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-26T06:24:36.305 INFO:teuthology.orchestra.run.smithi121.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-26T06:24:36.306 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-26T06:24:36.306 INFO:teuthology.orchestra.run.smithi121.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-26T06:24:36.306 INFO:teuthology.orchestra.run.smithi121.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-26T06:24:36.306 INFO:teuthology.orchestra.run.smithi121.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-26T06:24:36.306 INFO:teuthology.orchestra.run.smithi121.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-26T06:24:36.306 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-26T06:24:36.306 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-03-26T06:24:36.306 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T06:24:36.306 INFO:teuthology.orchestra.run.smithi121.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-26T06:24:36.306 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T06:24:36.307 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 20 M 2024-03-26T06:24:36.307 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 34 k 2024-03-26T06:24:36.307 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 2.1 M 2024-03-26T06:24:36.307 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 248 k 2024-03-26T06:24:36.307 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 4.7 M 2024-03-26T06:24:36.307 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 17 M 2024-03-26T06:24:36.307 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 16 k 2024-03-26T06:24:36.307 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 25 k 2024-03-26T06:24:36.307 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T06:24:36.307 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T06:24:36.307 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 163 k 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T06:24:36.308 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 503 k 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 5.1 M 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T06:24:36.309 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 45 k 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 131 k 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T06:24:36.310 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T06:24:36.311 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T06:24:36.312 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T06:24:36.313 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T06:24:36.314 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-03-26T06:24:36.315 INFO:teuthology.orchestra.run.smithi121.stdout:Install 150 Packages 2024-03-26T06:24:36.316 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 24 Packages 2024-03-26T06:24:36.316 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:24:36.316 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 223 M 2024-03-26T06:24:36.316 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-03-26T06:24:37.337 INFO:teuthology.orchestra.run.smithi121.stdout:(1/174): ceph-19.0.0-1964.g7e98f5f6.el9.x86_64. 36 kB/s | 6.6 kB 00:00 2024-03-26T06:24:37.546 INFO:teuthology.orchestra.run.smithi121.stdout:(2/174): ceph-fuse-19.0.0-1964.g7e98f5f6.el9.x8 4.2 MB/s | 889 kB 00:00 2024-03-26T06:24:37.612 INFO:teuthology.orchestra.run.smithi121.stdout:(3/174): ceph-immutable-object-cache-19.0.0-196 2.1 MB/s | 145 kB 00:00 2024-03-26T06:24:37.921 INFO:teuthology.orchestra.run.smithi121.stdout:(4/174): ceph-base-19.0.0-1964.g7e98f5f6.el9.x8 7.0 MB/s | 5.4 MB 00:00 2024-03-26T06:24:37.971 INFO:teuthology.orchestra.run.smithi121.stdout:(5/174): ceph-mds-19.0.0-1964.g7e98f5f6.el9.x86 5.9 MB/s | 2.1 MB 00:00 2024-03-26T06:24:38.071 INFO:teuthology.orchestra.run.smithi121.stdout:(6/174): ceph-mgr-19.0.0-1964.g7e98f5f6.el9.x86 9.9 MB/s | 1.5 MB 00:00 2024-03-26T06:24:38.497 INFO:teuthology.orchestra.run.smithi121.stdout:(7/174): ceph-mon-19.0.0-1964.g7e98f5f6.el9.x86 8.9 MB/s | 4.7 MB 00:00 2024-03-26T06:24:39.505 INFO:teuthology.orchestra.run.smithi121.stdout:(8/174): ceph-osd-19.0.0-1964.g7e98f5f6.el9.x86 12 MB/s | 17 MB 00:01 2024-03-26T06:24:39.614 INFO:teuthology.orchestra.run.smithi121.stdout:(9/174): ceph-radosgw-19.0.0-1964.g7e98f5f6.el9 8.8 MB/s | 9.8 MB 00:01 2024-03-26T06:24:39.639 INFO:teuthology.orchestra.run.smithi121.stdout:(10/174): ceph-selinux-19.0.0-1964.g7e98f5f6.el 189 kB/s | 25 kB 00:00 2024-03-26T06:24:39.714 INFO:teuthology.orchestra.run.smithi121.stdout:(11/174): cephfs-mirror-19.0.0-1964.g7e98f5f6.e 2.9 MB/s | 219 kB 00:00 2024-03-26T06:24:39.781 INFO:teuthology.orchestra.run.smithi121.stdout:(12/174): cephfs-shell-19.0.0-1964.g7e98f5f6.el 405 kB/s | 27 kB 00:00 2024-03-26T06:24:39.848 INFO:teuthology.orchestra.run.smithi121.stdout:(13/174): libcephfs-devel-19.0.0-1964.g7e98f5f6 467 kB/s | 31 kB 00:00 2024-03-26T06:24:39.940 INFO:teuthology.orchestra.run.smithi121.stdout:(14/174): libcephfs2-19.0.0-1964.g7e98f5f6.el9. 8.0 MB/s | 745 kB 00:00 2024-03-26T06:24:40.007 INFO:teuthology.orchestra.run.smithi121.stdout:(15/174): libcephsqlite-19.0.0-1964.g7e98f5f6.e 2.4 MB/s | 163 kB 00:00 2024-03-26T06:24:40.074 INFO:teuthology.orchestra.run.smithi121.stdout:(16/174): librados-devel-19.0.0-1964.g7e98f5f6. 1.9 MB/s | 127 kB 00:00 2024-03-26T06:24:40.174 INFO:teuthology.orchestra.run.smithi121.stdout:(17/174): libradosstriper1-19.0.0-1964.g7e98f5f 4.9 MB/s | 503 kB 00:00 2024-03-26T06:24:40.296 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-26T06:24:40.658 INFO:teuthology.orchestra.run.smithi121.stdout:(18/174): librgw2-19.0.0-1964.g7e98f5f6.el9.x86 11 MB/s | 5.1 MB 00:00 2024-03-26T06:24:40.716 INFO:teuthology.orchestra.run.smithi121.stdout:(19/174): python3-ceph-argparse-19.0.0-1964.g7e 779 kB/s | 45 kB 00:00 2024-03-26T06:24:40.783 INFO:teuthology.orchestra.run.smithi121.stdout:(20/174): python3-ceph-common-19.0.0-1964.g7e98 1.9 MB/s | 131 kB 00:00 2024-03-26T06:24:40.859 INFO:teuthology.orchestra.run.smithi121.stdout:(21/174): python3-cephfs-19.0.0-1964.g7e98f5f6. 2.1 MB/s | 162 kB 00:00 2024-03-26T06:24:40.942 INFO:teuthology.orchestra.run.smithi121.stdout:(22/174): python3-rados-19.0.0-1964.g7e98f5f6.e 3.8 MB/s | 321 kB 00:00 2024-03-26T06:24:41.018 INFO:teuthology.orchestra.run.smithi121.stdout:(23/174): python3-rbd-19.0.0-1964.g7e98f5f6.el9 3.9 MB/s | 298 kB 00:00 2024-03-26T06:24:41.093 INFO:teuthology.orchestra.run.smithi121.stdout:(24/174): python3-rgw-19.0.0-1964.g7e98f5f6.el9 1.3 MB/s | 99 kB 00:00 2024-03-26T06:24:41.168 INFO:teuthology.orchestra.run.smithi121.stdout:(25/174): rbd-fuse-19.0.0-1964.g7e98f5f6.el9.x8 1.1 MB/s | 86 kB 00:00 2024-03-26T06:24:41.477 INFO:teuthology.orchestra.run.smithi121.stdout:(26/174): ceph-common-19.0.0-1964.g7e98f5f6.el9 4.6 MB/s | 20 MB 00:04 2024-03-26T06:24:41.544 INFO:teuthology.orchestra.run.smithi121.stdout:(27/174): rbd-mirror-19.0.0-1964.g7e98f5f6.el9. 8.2 MB/s | 3.1 MB 00:00 2024-03-26T06:24:41.569 INFO:teuthology.orchestra.run.smithi121.stdout:(28/174): rbd-nbd-19.0.0-1964.g7e98f5f6.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-26T06:24:41.611 INFO:teuthology.orchestra.run.smithi121.stdout:(29/174): ceph-grafana-dashboards-19.0.0-1964.g 508 kB/s | 34 kB 00:00 2024-03-26T06:24:41.636 INFO:teuthology.orchestra.run.smithi121.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1964.g7e98f5f 2.0 MB/s | 139 kB 00:00 2024-03-26T06:24:41.795 INFO:teuthology.orchestra.run.smithi121.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1964.g7e98f 9.8 MB/s | 1.8 MB 00:00 2024-03-26T06:24:41.862 INFO:teuthology.orchestra.run.smithi121.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1964.g7e 3.6 MB/s | 248 kB 00:00 2024-03-26T06:24:41.929 INFO:teuthology.orchestra.run.smithi121.stdout:(33/174): ceph-mgr-rook-19.0.0-1964.g7e98f5f6.e 739 kB/s | 49 kB 00:00 2024-03-26T06:24:41.996 INFO:teuthology.orchestra.run.smithi121.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1964.g7 246 kB/s | 16 kB 00:00 2024-03-26T06:24:42.079 INFO:teuthology.orchestra.run.smithi121.stdout:(35/174): ceph-volume-19.0.0-1964.g7e98f5f6.el9 3.1 MB/s | 264 kB 00:00 2024-03-26T06:24:42.179 INFO:teuthology.orchestra.run.smithi121.stdout:(36/174): cephadm-19.0.0-1964.g7e98f5f6.el9.noa 7.3 MB/s | 749 kB 00:00 2024-03-26T06:24:42.246 INFO:teuthology.orchestra.run.smithi121.stdout:(37/174): cephfs-top-19.0.0-1964.g7e98f5f6.el9. 273 kB/s | 18 kB 00:00 2024-03-26T06:24:42.455 INFO:teuthology.orchestra.run.smithi121.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.1 MB/s | 662 kB 00:00 2024-03-26T06:24:42.522 INFO:teuthology.orchestra.run.smithi121.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-26T06:24:42.560 INFO:teuthology.orchestra.run.smithi121.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.7 MB/s | 106 kB 00:00 2024-03-26T06:24:42.602 INFO:teuthology.orchestra.run.smithi121.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-26T06:24:42.627 INFO:teuthology.orchestra.run.smithi121.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 1.2 MB/s | 32 kB 00:00 2024-03-26T06:24:42.770 INFO:teuthology.orchestra.run.smithi121.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-03-26T06:24:42.809 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-26T06:24:42.810 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-26T06:24:42.825 INFO:teuthology.orchestra.run.smithi038.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-26T06:24:42.827 INFO:teuthology.orchestra.run.smithi038.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-26T06:24:42.828 INFO:teuthology.orchestra.run.smithi038.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-26T06:24:42.829 INFO:teuthology.orchestra.run.smithi038.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-26T06:24:42.831 INFO:teuthology.orchestra.run.smithi038.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-26T06:24:42.834 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-26T06:24:42.835 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-26T06:24:42.854 INFO:teuthology.orchestra.run.smithi121.stdout:(44/174): ceph-mgr-diskprediction-local-19.0.0- 6.1 MB/s | 7.4 MB 00:01 2024-03-26T06:24:42.879 INFO:teuthology.orchestra.run.smithi121.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 157 kB/s | 17 kB 00:00 2024-03-26T06:24:42.904 INFO:teuthology.orchestra.run.smithi121.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 202 kB/s | 10 kB 00:00 2024-03-26T06:24:42.921 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2024-03-26T06:24:42.929 INFO:teuthology.orchestra.run.smithi121.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-26T06:24:42.933 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-03-26T06:24:42.934 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2024-03-26T06:24:42.934 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-03-26T06:24:42.934 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2024-03-26T06:24:42.934 INFO:teuthology.orchestra.run.smithi038.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-26T06:24:42.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 6.6 k 2024-03-26T06:24:42.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 5.4 M 2024-03-26T06:24:42.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 889 k 2024-03-26T06:24:42.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 145 k 2024-03-26T06:24:42.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 1.5 M 2024-03-26T06:24:42.934 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 139 k 2024-03-26T06:24:42.935 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 1.8 M 2024-03-26T06:24:42.935 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 7.4 M 2024-03-26T06:24:42.935 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 49 k 2024-03-26T06:24:42.935 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 9.8 M 2024-03-26T06:24:42.935 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 47 M 2024-03-26T06:24:42.935 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 264 k 2024-03-26T06:24:42.935 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 749 k 2024-03-26T06:24:42.935 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 219 k 2024-03-26T06:24:42.935 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 27 k 2024-03-26T06:24:42.935 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 18 k 2024-03-26T06:24:42.935 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 31 k 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 745 k 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 127 k 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-26T06:24:42.936 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 162 k 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 321 k 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 298 k 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 99 k 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 86 k 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 3.1 M 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 170 k 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-26T06:24:42.937 INFO:teuthology.orchestra.run.smithi038.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-26T06:24:42.938 INFO:teuthology.orchestra.run.smithi038.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-26T06:24:42.938 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-26T06:24:42.938 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-26T06:24:42.938 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-26T06:24:42.938 INFO:teuthology.orchestra.run.smithi038.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-26T06:24:42.938 INFO:teuthology.orchestra.run.smithi038.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-26T06:24:42.938 INFO:teuthology.orchestra.run.smithi038.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-26T06:24:42.938 INFO:teuthology.orchestra.run.smithi038.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-26T06:24:42.938 INFO:teuthology.orchestra.run.smithi038.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-26T06:24:42.938 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 3.4 M 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 3.1 M 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2024-03-26T06:24:42.939 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T06:24:42.940 INFO:teuthology.orchestra.run.smithi038.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-26T06:24:42.940 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T06:24:42.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 20 M 2024-03-26T06:24:42.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 34 k 2024-03-26T06:24:42.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 2.1 M 2024-03-26T06:24:42.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 248 k 2024-03-26T06:24:42.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 4.7 M 2024-03-26T06:24:42.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 17 M 2024-03-26T06:24:42.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 16 k 2024-03-26T06:24:42.940 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 25 k 2024-03-26T06:24:42.940 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T06:24:42.941 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T06:24:42.941 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T06:24:42.941 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T06:24:42.941 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T06:24:42.941 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T06:24:42.941 INFO:teuthology.orchestra.run.smithi038.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T06:24:42.941 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T06:24:42.941 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T06:24:42.941 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 163 k 2024-03-26T06:24:42.941 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T06:24:42.941 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 503 k 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 5.1 M 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T06:24:42.942 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T06:24:42.943 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T06:24:42.943 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T06:24:42.943 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T06:24:42.943 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T06:24:42.943 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-26T06:24:42.943 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T06:24:42.943 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T06:24:42.943 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T06:24:42.943 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-26T06:24:42.943 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T06:24:42.943 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 45 k 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 131 k 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T06:24:42.944 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T06:24:42.945 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T06:24:42.946 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T06:24:42.946 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T06:24:42.946 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T06:24:42.946 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T06:24:42.946 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T06:24:42.946 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T06:24:42.946 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T06:24:42.946 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T06:24:42.946 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T06:24:42.946 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T06:24:42.946 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T06:24:42.947 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T06:24:42.948 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout:Installing weak dependencies: 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2024-03-26T06:24:42.949 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================ 2024-03-26T06:24:42.950 INFO:teuthology.orchestra.run.smithi038.stdout:Install 150 Packages 2024-03-26T06:24:42.950 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 24 Packages 2024-03-26T06:24:42.950 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:24:42.950 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 223 M 2024-03-26T06:24:42.950 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2024-03-26T06:24:42.955 INFO:teuthology.orchestra.run.smithi121.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 879 kB/s | 44 kB 00:00 2024-03-26T06:24:43.380 INFO:teuthology.orchestra.run.smithi121.stdout:(49/174): ceph-test-19.0.0-1964.g7e98f5f6.el9.x 13 MB/s | 47 MB 00:03 2024-03-26T06:24:43.422 INFO:teuthology.orchestra.run.smithi121.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 1.3 MB/s | 661 kB 00:00 2024-03-26T06:24:43.447 INFO:teuthology.orchestra.run.smithi121.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 1.1 MB/s | 578 kB 00:00 2024-03-26T06:24:43.472 INFO:teuthology.orchestra.run.smithi121.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 517 kB/s | 46 kB 00:00 2024-03-26T06:24:43.498 INFO:teuthology.orchestra.run.smithi121.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-26T06:24:43.523 INFO:teuthology.orchestra.run.smithi121.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-26T06:24:43.548 INFO:teuthology.orchestra.run.smithi121.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-26T06:24:43.573 INFO:teuthology.orchestra.run.smithi121.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 688 kB/s | 34 kB 00:00 2024-03-26T06:24:43.664 INFO:teuthology.orchestra.run.smithi038.stdout:(1/174): ceph-19.0.0-1964.g7e98f5f6.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-03-26T06:24:43.673 INFO:teuthology.orchestra.run.smithi121.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 32 MB/s | 4.7 MB 00:00 2024-03-26T06:24:43.707 INFO:teuthology.orchestra.run.smithi121.stdout:(58/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-26T06:24:43.732 INFO:teuthology.orchestra.run.smithi121.stdout:(59/174): python3-attrs-20.3.0-7.el9.noarch.rpm 497 kB/s | 78 kB 00:00 2024-03-26T06:24:43.856 INFO:teuthology.orchestra.run.smithi038.stdout:(2/174): ceph-fuse-19.0.0-1964.g7e98f5f6.el9.x8 4.5 MB/s | 889 kB 00:00 2024-03-26T06:24:43.857 INFO:teuthology.orchestra.run.smithi121.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 32 MB/s | 5.8 MB 00:00 2024-03-26T06:24:43.883 INFO:teuthology.orchestra.run.smithi121.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 166 kB/s | 24 kB 00:00 2024-03-26T06:24:43.900 INFO:teuthology.orchestra.run.smithi121.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 110 kB/s | 18 kB 00:00 2024-03-26T06:24:43.906 INFO:teuthology.orchestra.run.smithi038.stdout:(3/174): ceph-immutable-object-cache-19.0.0-196 2.8 MB/s | 145 kB 00:00 2024-03-26T06:24:43.925 INFO:teuthology.orchestra.run.smithi121.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-26T06:24:43.951 INFO:teuthology.orchestra.run.smithi121.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 633 kB/s | 32 kB 00:00 2024-03-26T06:24:44.050 INFO:teuthology.orchestra.run.smithi121.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-26T06:24:44.056 INFO:teuthology.orchestra.run.smithi038.stdout:(4/174): ceph-mds-19.0.0-1964.g7e98f5f6.el9.x86 14 MB/s | 2.1 MB 00:00 2024-03-26T06:24:44.076 INFO:teuthology.orchestra.run.smithi121.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-26T06:24:44.101 INFO:teuthology.orchestra.run.smithi121.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-26T06:24:44.118 INFO:teuthology.orchestra.run.smithi121.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-26T06:24:44.135 INFO:teuthology.orchestra.run.smithi121.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-26T06:24:44.160 INFO:teuthology.orchestra.run.smithi121.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 4.0 MB/s | 238 kB 00:00 2024-03-26T06:24:44.185 INFO:teuthology.orchestra.run.smithi121.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-26T06:24:44.199 INFO:teuthology.orchestra.run.smithi038.stdout:(5/174): ceph-mgr-19.0.0-1964.g7e98f5f6.el9.x86 11 MB/s | 1.5 MB 00:00 2024-03-26T06:24:44.210 INFO:teuthology.orchestra.run.smithi121.stdout:(72/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-26T06:24:44.227 INFO:teuthology.orchestra.run.smithi121.stdout:(73/174): python3-pytz-2021.1-5.el9.noarch.rpm 711 kB/s | 47 kB 00:00 2024-03-26T06:24:44.246 INFO:teuthology.orchestra.run.smithi121.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 738 kB/s | 43 kB 00:00 2024-03-26T06:24:44.607 INFO:teuthology.orchestra.run.smithi038.stdout:(6/174): ceph-mon-19.0.0-1964.g7e98f5f6.el9.x86 15 MB/s | 4.7 MB 00:00 2024-03-26T06:24:44.607 INFO:teuthology.orchestra.run.smithi121.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 579 kB/s | 39 kB 00:00 2024-03-26T06:24:44.607 INFO:teuthology.orchestra.run.smithi121.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 259 kB/s | 41 kB 00:00 2024-03-26T06:24:44.612 INFO:teuthology.orchestra.run.smithi121.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-26T06:24:44.637 INFO:teuthology.orchestra.run.smithi121.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 882 kB/s | 300 kB 00:00 2024-03-26T06:24:44.679 INFO:teuthology.orchestra.run.smithi121.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 206 kB/s | 56 kB 00:00 2024-03-26T06:24:44.696 INFO:teuthology.orchestra.run.smithi121.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 754 kB/s | 63 kB 00:00 2024-03-26T06:24:44.713 INFO:teuthology.orchestra.run.smithi121.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-26T06:24:44.738 INFO:teuthology.orchestra.run.smithi121.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-26T06:24:44.763 INFO:teuthology.orchestra.run.smithi121.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-26T06:24:44.788 INFO:teuthology.orchestra.run.smithi121.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-26T06:24:44.822 INFO:teuthology.orchestra.run.smithi121.stdout:(85/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-26T06:24:44.847 INFO:teuthology.orchestra.run.smithi121.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-26T06:24:44.849 INFO:teuthology.orchestra.run.smithi038.stdout:(7/174): ceph-base-19.0.0-1964.g7e98f5f6.el9.x8 3.9 MB/s | 5.4 MB 00:01 2024-03-26T06:24:44.873 INFO:teuthology.orchestra.run.smithi121.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-26T06:24:45.525 INFO:teuthology.orchestra.run.smithi038.stdout:(8/174): ceph-osd-19.0.0-1964.g7e98f5f6.el9.x86 17 MB/s | 17 MB 00:01 2024-03-26T06:24:45.677 INFO:teuthology.orchestra.run.smithi038.stdout:(9/174): ceph-selinux-19.0.0-1964.g7e98f5f6.el9 168 kB/s | 25 kB 00:00 2024-03-26T06:24:45.706 INFO:teuthology.orchestra.run.smithi121.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 234 kB/s | 195 kB 00:00 2024-03-26T06:24:45.731 INFO:teuthology.orchestra.run.smithi121.stdout:(89/174): jq-1.6-16.el9.x86_64.rpm 212 kB/s | 187 kB 00:00 2024-03-26T06:24:45.807 INFO:teuthology.orchestra.run.smithi121.stdout:(90/174): elfutils-libelf-devel-0.190-2.el9.x86 287 kB/s | 21 kB 00:00 2024-03-26T06:24:45.857 INFO:teuthology.orchestra.run.smithi121.stdout:(91/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 510 kB/s | 25 kB 00:00 2024-03-26T06:24:46.049 INFO:teuthology.orchestra.run.smithi121.stdout:(92/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 22 MB/s | 4.1 MB 00:00 2024-03-26T06:24:46.107 INFO:teuthology.orchestra.run.smithi121.stdout:(93/174): postgresql-13.14-1.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-03-26T06:24:46.158 INFO:teuthology.orchestra.run.smithi121.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 18 MB/s | 911 kB 00:00 2024-03-26T06:24:46.200 INFO:teuthology.orchestra.run.smithi121.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 3.1 MB/s | 133 kB 00:00 2024-03-26T06:24:46.325 INFO:teuthology.orchestra.run.smithi121.stdout:(96/174): postgresql-server-13.14-1.el9.x86_64. 46 MB/s | 5.8 MB 00:00 2024-03-26T06:24:46.368 INFO:teuthology.orchestra.run.smithi121.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 5.6 MB/s | 245 kB 00:00 2024-03-26T06:24:46.410 INFO:teuthology.orchestra.run.smithi121.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 5.8 MB/s | 248 kB 00:00 2024-03-26T06:24:46.452 INFO:teuthology.orchestra.run.smithi121.stdout:(99/174): qatlib-24.02.0-1.el9.x86_64.rpm 5.2 MB/s | 221 kB 00:00 2024-03-26T06:24:46.493 INFO:teuthology.orchestra.run.smithi121.stdout:(100/174): qatlib-service-24.02.0-1.el9.x86_64. 866 kB/s | 36 kB 00:00 2024-03-26T06:24:46.536 INFO:teuthology.orchestra.run.smithi121.stdout:(101/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-03-26T06:24:46.569 INFO:teuthology.orchestra.run.smithi121.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.8 MB/s | 94 kB 00:00 2024-03-26T06:24:46.603 INFO:teuthology.orchestra.run.smithi121.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 667 kB/s | 22 kB 00:00 2024-03-26T06:24:46.653 INFO:teuthology.orchestra.run.smithi121.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 17 MB/s | 863 kB 00:00 2024-03-26T06:24:46.970 INFO:teuthology.orchestra.run.smithi121.stdout:(105/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-03-26T06:24:47.037 INFO:teuthology.orchestra.run.smithi121.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 4.5 MB/s | 309 kB 00:00 2024-03-26T06:24:47.112 INFO:teuthology.orchestra.run.smithi121.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-26T06:24:47.313 INFO:teuthology.orchestra.run.smithi121.stdout:(108/174): libarrow-9.0.0-10.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2024-03-26T06:24:47.371 INFO:teuthology.orchestra.run.smithi121.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-03-26T06:24:47.430 INFO:teuthology.orchestra.run.smithi121.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2024-03-26T06:24:47.488 INFO:teuthology.orchestra.run.smithi121.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-26T06:24:47.564 INFO:teuthology.orchestra.run.smithi121.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-03-26T06:24:47.647 INFO:teuthology.orchestra.run.smithi121.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-26T06:24:47.761 INFO:teuthology.orchestra.run.smithi121.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 4.7 MB/s | 548 kB 00:00 2024-03-26T06:24:47.822 INFO:teuthology.orchestra.run.smithi121.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 712 kB/s | 43 kB 00:00 2024-03-26T06:24:47.881 INFO:teuthology.orchestra.run.smithi121.stdout:(116/174): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-03-26T06:24:47.940 INFO:teuthology.orchestra.run.smithi121.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-03-26T06:24:48.007 INFO:teuthology.orchestra.run.smithi121.stdout:(118/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-03-26T06:24:48.065 INFO:teuthology.orchestra.run.smithi121.stdout:(119/174): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-03-26T06:24:48.132 INFO:teuthology.orchestra.run.smithi121.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-03-26T06:24:48.199 INFO:teuthology.orchestra.run.smithi121.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 530 kB/s | 35 kB 00:00 2024-03-26T06:24:48.266 INFO:teuthology.orchestra.run.smithi121.stdout:(122/174): python3-google-auth-2.28.2-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-26T06:24:48.325 INFO:teuthology.orchestra.run.smithi121.stdout:(123/174): python3-requests-2.25.1-8.el9.noarch 48 kB/s | 125 kB 00:02 2024-03-26T06:24:48.392 INFO:teuthology.orchestra.run.smithi121.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 16 MB/s | 2.0 MB 00:00 2024-03-26T06:24:48.450 INFO:teuthology.orchestra.run.smithi121.stdout:(125/174): libgfortran-11.4.1-3.el9.x86_64.rpm 223 kB/s | 803 kB 00:03 2024-03-26T06:24:48.475 INFO:teuthology.orchestra.run.smithi121.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-03-26T06:24:48.501 INFO:teuthology.orchestra.run.smithi121.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-03-26T06:24:48.559 INFO:teuthology.orchestra.run.smithi121.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 334 kB/s | 19 kB 00:00 2024-03-26T06:24:48.584 INFO:teuthology.orchestra.run.smithi121.stdout:(129/174): python3-grpcio-tools-1.46.7-10.el9.x 556 kB/s | 144 kB 00:00 2024-03-26T06:24:48.610 INFO:teuthology.orchestra.run.smithi121.stdout:(130/174): python3-jaraco-collections-3.0.0-8.e 173 kB/s | 23 kB 00:00 2024-03-26T06:24:48.635 INFO:teuthology.orchestra.run.smithi121.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-03-26T06:24:48.656 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - AppStream 1.5 MB/s | 19 MB 00:12 2024-03-26T06:24:48.702 INFO:teuthology.orchestra.run.smithi121.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-03-26T06:24:48.727 INFO:teuthology.orchestra.run.smithi121.stdout:(133/174): python3-logutils-0.3.5-21.el9.noarch 395 kB/s | 46 kB 00:00 2024-03-26T06:24:48.761 INFO:teuthology.orchestra.run.smithi121.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-03-26T06:24:48.786 INFO:teuthology.orchestra.run.smithi121.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 688 kB/s | 58 kB 00:00 2024-03-26T06:24:48.828 INFO:teuthology.orchestra.run.smithi121.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-03-26T06:24:48.853 INFO:teuthology.orchestra.run.smithi121.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-03-26T06:24:48.887 INFO:teuthology.orchestra.run.smithi121.stdout:(138/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-03-26T06:24:48.912 INFO:teuthology.orchestra.run.smithi121.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 336 kB/s | 28 kB 00:00 2024-03-26T06:24:49.016 INFO:teuthology.orchestra.run.smithi038.stdout:(10/174): ceph-test-19.0.0-1964.g7e98f5f6.el9.x 14 MB/s | 47 MB 00:03 2024-03-26T06:24:49.017 INFO:teuthology.orchestra.run.smithi121.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 188 kB/s | 31 kB 00:00 2024-03-26T06:24:49.097 INFO:teuthology.orchestra.run.smithi121.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 321 kB/s | 59 kB 00:00 2024-03-26T06:24:49.122 INFO:teuthology.orchestra.run.smithi121.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 344 kB/s | 36 kB 00:00 2024-03-26T06:24:49.147 INFO:teuthology.orchestra.run.smithi121.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 726 kB/s | 188 kB 00:00 2024-03-26T06:24:49.172 INFO:teuthology.orchestra.run.smithi121.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 677 kB/s | 51 kB 00:00 2024-03-26T06:24:49.198 INFO:teuthology.orchestra.run.smithi121.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-26T06:24:49.209 INFO:teuthology.orchestra.run.smithi038.stdout:(11/174): ceph-radosgw-19.0.0-1964.g7e98f5f6.el 2.2 MB/s | 9.8 MB 00:04 2024-03-26T06:24:49.223 INFO:teuthology.orchestra.run.smithi121.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-26T06:24:49.256 INFO:teuthology.orchestra.run.smithi121.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-03-26T06:24:49.273 INFO:teuthology.orchestra.run.smithi121.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 264 kB/s | 20 kB 00:00 2024-03-26T06:24:49.299 INFO:teuthology.orchestra.run.smithi121.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-26T06:24:49.317 INFO:teuthology.orchestra.run.smithi038.stdout:(12/174): cephfs-mirror-19.0.0-1964.g7e98f5f6.e 732 kB/s | 219 kB 00:00 2024-03-26T06:24:49.342 INFO:teuthology.orchestra.run.smithi038.stdout:(13/174): cephfs-shell-19.0.0-1964.g7e98f5f6.el 203 kB/s | 27 kB 00:00 2024-03-26T06:24:49.368 INFO:teuthology.orchestra.run.smithi038.stdout:(14/174): libcephfs-devel-19.0.0-1964.g7e98f5f6 619 kB/s | 31 kB 00:00 2024-03-26T06:24:49.373 INFO:teuthology.orchestra.run.smithi121.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-26T06:24:49.416 INFO:teuthology.orchestra.run.smithi121.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-26T06:24:49.418 INFO:teuthology.orchestra.run.smithi038.stdout:(15/174): libcephsqlite-19.0.0-1964.g7e98f5f6.e 3.2 MB/s | 163 kB 00:00 2024-03-26T06:24:49.502 INFO:teuthology.orchestra.run.smithi038.stdout:(16/174): librados-devel-19.0.0-1964.g7e98f5f6. 1.5 MB/s | 127 kB 00:00 2024-03-26T06:24:49.757 INFO:teuthology.orchestra.run.smithi121.stdout:(152/174): librados2-19.0.0-1964.g7e98f5f6.el9. 7.0 MB/s | 3.4 MB 00:00 2024-03-26T06:24:49.844 INFO:teuthology.orchestra.run.smithi038.stdout:(17/174): libcephfs2-19.0.0-1964.g7e98f5f6.el9. 1.5 MB/s | 745 kB 00:00 2024-03-26T06:24:49.858 INFO:teuthology.orchestra.run.smithi121.stdout:(153/174): librbd1-19.0.0-1964.g7e98f5f6.el9.x8 5.6 MB/s | 3.1 MB 00:00 2024-03-26T06:24:49.877 INFO:teuthology.orchestra.run.smithi038.stdout:(18/174): libradosstriper1-19.0.0-1964.g7e98f5f 1.3 MB/s | 503 kB 00:00 2024-03-26T06:24:49.919 INFO:teuthology.orchestra.run.smithi038.stdout:(19/174): python3-ceph-argparse-19.0.0-1964.g7e 1.1 MB/s | 45 kB 00:00 2024-03-26T06:24:49.970 INFO:teuthology.orchestra.run.smithi038.stdout:(20/174): python3-ceph-common-19.0.0-1964.g7e98 2.6 MB/s | 131 kB 00:00 2024-03-26T06:24:50.011 INFO:teuthology.orchestra.run.smithi038.stdout:(21/174): python3-cephfs-19.0.0-1964.g7e98f5f6. 3.8 MB/s | 162 kB 00:00 2024-03-26T06:24:50.070 INFO:teuthology.orchestra.run.smithi038.stdout:(22/174): python3-rados-19.0.0-1964.g7e98f5f6.e 5.4 MB/s | 321 kB 00:00 2024-03-26T06:24:50.129 INFO:teuthology.orchestra.run.smithi038.stdout:(23/174): python3-rbd-19.0.0-1964.g7e98f5f6.el9 5.0 MB/s | 298 kB 00:00 2024-03-26T06:24:50.179 INFO:teuthology.orchestra.run.smithi038.stdout:(24/174): python3-rgw-19.0.0-1964.g7e98f5f6.el9 1.9 MB/s | 99 kB 00:00 2024-03-26T06:24:50.213 INFO:teuthology.orchestra.run.smithi038.stdout:(25/174): rbd-fuse-19.0.0-1964.g7e98f5f6.el9.x8 2.5 MB/s | 86 kB 00:00 2024-03-26T06:24:50.447 INFO:teuthology.orchestra.run.smithi038.stdout:(26/174): rbd-mirror-19.0.0-1964.g7e98f5f6.el9. 13 MB/s | 3.1 MB 00:00 2024-03-26T06:24:50.489 INFO:teuthology.orchestra.run.smithi038.stdout:(27/174): rbd-nbd-19.0.0-1964.g7e98f5f6.el9.x86 4.0 MB/s | 170 kB 00:00 2024-03-26T06:24:50.539 INFO:teuthology.orchestra.run.smithi038.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1964.g 682 kB/s | 34 kB 00:00 2024-03-26T06:24:50.590 INFO:teuthology.orchestra.run.smithi038.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1964.g7e98f5f 2.7 MB/s | 139 kB 00:00 2024-03-26T06:24:50.814 INFO:teuthology.orchestra.run.smithi038.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1964.g7e98f 8.0 MB/s | 1.8 MB 00:00 2024-03-26T06:24:51.015 INFO:teuthology.orchestra.run.smithi038.stdout:(31/174): ceph-common-19.0.0-1964.g7e98f5f6.el9 2.7 MB/s | 20 MB 00:07 2024-03-26T06:24:51.190 INFO:teuthology.orchestra.run.smithi038.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1964.g7e 1.4 MB/s | 248 kB 00:00 2024-03-26T06:24:51.282 INFO:teuthology.orchestra.run.smithi038.stdout:(33/174): ceph-mgr-rook-19.0.0-1964.g7e98f5f6.e 538 kB/s | 49 kB 00:00 2024-03-26T06:24:51.382 INFO:teuthology.orchestra.run.smithi038.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1964.g7 164 kB/s | 16 kB 00:00 2024-03-26T06:24:51.582 INFO:teuthology.orchestra.run.smithi038.stdout:(35/174): ceph-volume-19.0.0-1964.g7e98f5f6.el9 1.3 MB/s | 264 kB 00:00 2024-03-26T06:24:51.774 INFO:teuthology.orchestra.run.smithi038.stdout:(36/174): ceph-mgr-diskprediction-local-19.0.0- 7.7 MB/s | 7.4 MB 00:00 2024-03-26T06:24:51.833 INFO:teuthology.orchestra.run.smithi038.stdout:(37/174): cephfs-top-19.0.0-1964.g7e98f5f6.el9. 314 kB/s | 18 kB 00:00 2024-03-26T06:24:51.933 INFO:teuthology.orchestra.run.smithi038.stdout:(38/174): cephadm-19.0.0-1964.g7e98f5f6.el9.noa 2.1 MB/s | 749 kB 00:00 2024-03-26T06:24:52.067 INFO:teuthology.orchestra.run.smithi038.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 662 kB 00:00 2024-03-26T06:24:52.092 INFO:teuthology.orchestra.run.smithi038.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 5.7 MB/s | 920 kB 00:00 2024-03-26T06:24:52.109 INFO:teuthology.orchestra.run.smithi038.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-26T06:24:52.134 INFO:teuthology.orchestra.run.smithi038.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-26T06:24:52.160 INFO:teuthology.orchestra.run.smithi038.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 639 kB/s | 32 kB 00:00 2024-03-26T06:24:52.285 INFO:teuthology.orchestra.run.smithi038.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-03-26T06:24:52.319 INFO:teuthology.orchestra.run.smithi038.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 109 kB/s | 17 kB 00:00 2024-03-26T06:24:52.352 INFO:teuthology.orchestra.run.smithi038.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 306 kB/s | 10 kB 00:00 2024-03-26T06:24:52.378 INFO:teuthology.orchestra.run.smithi038.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-26T06:24:52.403 INFO:teuthology.orchestra.run.smithi038.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-26T06:24:52.428 INFO:teuthology.orchestra.run.smithi038.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-26T06:24:52.462 INFO:teuthology.orchestra.run.smithi038.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.9 MB/s | 578 kB 00:00 2024-03-26T06:24:52.529 INFO:teuthology.orchestra.run.smithi038.stdout:(51/174): librgw2-19.0.0-1964.g7e98f5f6.el9.x86 1.9 MB/s | 5.1 MB 00:02 2024-03-26T06:24:52.554 INFO:teuthology.orchestra.run.smithi038.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 372 kB/s | 46 kB 00:00 2024-03-26T06:24:52.574 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-26T06:24:52.579 INFO:teuthology.orchestra.run.smithi038.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 282 kB 00:00 2024-03-26T06:24:52.596 INFO:teuthology.orchestra.run.smithi038.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-26T06:24:52.621 INFO:teuthology.orchestra.run.smithi038.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.2 MB/s | 217 kB 00:00 2024-03-26T06:24:52.647 INFO:teuthology.orchestra.run.smithi038.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 511 kB/s | 34 kB 00:00 2024-03-26T06:24:52.697 INFO:teuthology.orchestra.run.smithi038.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-03-26T06:24:52.789 INFO:teuthology.orchestra.run.smithi038.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-03-26T06:24:52.806 INFO:teuthology.orchestra.run.smithi038.stdout:(59/174): python3-attrs-20.3.0-7.el9.noarch.rpm 492 kB/s | 78 kB 00:00 2024-03-26T06:24:52.923 INFO:teuthology.orchestra.run.smithi038.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-26T06:24:52.948 INFO:teuthology.orchestra.run.smithi038.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 158 kB/s | 24 kB 00:00 2024-03-26T06:24:52.973 INFO:teuthology.orchestra.run.smithi038.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 110 kB/s | 18 kB 00:00 2024-03-26T06:24:52.998 INFO:teuthology.orchestra.run.smithi038.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-26T06:24:53.024 INFO:teuthology.orchestra.run.smithi038.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-26T06:24:53.124 INFO:teuthology.orchestra.run.smithi038.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-26T06:24:53.149 INFO:teuthology.orchestra.run.smithi038.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-26T06:24:53.175 INFO:teuthology.orchestra.run.smithi038.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-26T06:24:53.200 INFO:teuthology.orchestra.run.smithi038.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 919 kB/s | 69 kB 00:00 2024-03-26T06:24:53.225 INFO:teuthology.orchestra.run.smithi038.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 826 kB/s | 41 kB 00:00 2024-03-26T06:24:53.250 INFO:teuthology.orchestra.run.smithi038.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-26T06:24:53.275 INFO:teuthology.orchestra.run.smithi038.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-26T06:24:53.300 INFO:teuthology.orchestra.run.smithi038.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 952 kB/s | 47 kB 00:00 2024-03-26T06:24:53.326 INFO:teuthology.orchestra.run.smithi038.stdout:(73/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-26T06:24:53.351 INFO:teuthology.orchestra.run.smithi038.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-03-26T06:24:53.432 INFO:teuthology.orchestra.run.smithi121.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 12 kB/s | 42 kB 00:03 2024-03-26T06:24:53.668 INFO:teuthology.orchestra.run.smithi038.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 123 kB/s | 39 kB 00:00 2024-03-26T06:24:53.693 INFO:teuthology.orchestra.run.smithi038.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 121 kB/s | 41 kB 00:00 2024-03-26T06:24:53.860 INFO:teuthology.orchestra.run.smithi038.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-03-26T06:24:53.886 INFO:teuthology.orchestra.run.smithi038.stdout:(78/174): uuid-1.6.2-55.el9.x86_64.rpm 293 kB/s | 56 kB 00:00 2024-03-26T06:24:53.911 INFO:teuthology.orchestra.run.smithi038.stdout:(79/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-03-26T06:24:53.936 INFO:teuthology.orchestra.run.smithi038.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 833 kB/s | 63 kB 00:00 2024-03-26T06:24:53.961 INFO:teuthology.orchestra.run.smithi038.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 590 kB/s | 44 kB 00:00 2024-03-26T06:24:53.986 INFO:teuthology.orchestra.run.smithi038.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-26T06:24:54.012 INFO:teuthology.orchestra.run.smithi038.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-26T06:24:54.054 INFO:teuthology.orchestra.run.smithi038.stdout:(84/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-26T06:24:54.079 INFO:teuthology.orchestra.run.smithi038.stdout:(85/174): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 103 kB 00:00 2024-03-26T06:24:54.104 INFO:teuthology.orchestra.run.smithi038.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-26T06:24:54.129 INFO:teuthology.orchestra.run.smithi038.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-26T06:24:54.288 INFO:teuthology.orchestra.run.smithi038.stdout:(88/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-03-26T06:24:54.313 INFO:teuthology.orchestra.run.smithi038.stdout:(89/174): jq-1.6-16.el9.x86_64.rpm 898 kB/s | 187 kB 00:00 2024-03-26T06:24:54.339 INFO:teuthology.orchestra.run.smithi038.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-26T06:24:54.356 INFO:teuthology.orchestra.run.smithi038.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 513 kB/s | 21 kB 00:00 2024-03-26T06:24:54.372 INFO:teuthology.orchestra.run.smithi038.stdout:(92/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 760 kB/s | 25 kB 00:00 2024-03-26T06:24:54.406 INFO:teuthology.orchestra.run.smithi038.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 803 kB 00:00 2024-03-26T06:24:54.490 INFO:teuthology.orchestra.run.smithi038.stdout:(94/174): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-03-26T06:24:54.532 INFO:teuthology.orchestra.run.smithi038.stdout:(95/174): postgresql-13.14-1.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-26T06:24:54.557 INFO:teuthology.orchestra.run.smithi038.stdout:(96/174): postgresql-private-libs-13.14-1.el9.x 1.9 MB/s | 133 kB 00:00 2024-03-26T06:24:54.633 INFO:teuthology.orchestra.run.smithi038.stdout:(97/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 15 MB/s | 4.1 MB 00:00 2024-03-26T06:24:54.658 INFO:teuthology.orchestra.run.smithi038.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-26T06:24:54.685 INFO:teuthology.orchestra.run.smithi038.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-03-26T06:24:54.708 INFO:teuthology.orchestra.run.smithi038.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-03-26T06:24:54.725 INFO:teuthology.orchestra.run.smithi038.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 860 kB/s | 36 kB 00:00 2024-03-26T06:24:54.742 INFO:teuthology.orchestra.run.smithi038.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-03-26T06:24:54.776 INFO:teuthology.orchestra.run.smithi038.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-26T06:24:54.859 INFO:teuthology.orchestra.run.smithi038.stdout:(104/174): postgresql-server-13.14-1.el9.x86_64 18 MB/s | 5.8 MB 00:00 2024-03-26T06:24:54.918 INFO:teuthology.orchestra.run.smithi038.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 127 kB/s | 22 kB 00:00 2024-03-26T06:24:54.966 INFO:teuthology.orchestra.run.smithi170.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-26T06:24:54.993 INFO:teuthology.orchestra.run.smithi038.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 4.0 MB/s | 548 kB 00:00 2024-03-26T06:24:55.018 INFO:teuthology.orchestra.run.smithi038.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-03-26T06:24:55.044 INFO:teuthology.orchestra.run.smithi038.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 385 kB/s | 19 kB 00:00 2024-03-26T06:24:55.077 INFO:teuthology.orchestra.run.smithi038.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 788 kB/s | 26 kB 00:00 2024-03-26T06:24:55.111 INFO:teuthology.orchestra.run.smithi038.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-26T06:24:55.136 INFO:teuthology.orchestra.run.smithi038.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-03-26T06:24:55.203 INFO:teuthology.orchestra.run.smithi038.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-03-26T06:24:55.220 INFO:teuthology.orchestra.run.smithi038.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-26T06:24:55.253 INFO:teuthology.orchestra.run.smithi038.stdout:(114/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.8 MB/s | 863 kB 00:00 2024-03-26T06:24:55.296 INFO:teuthology.orchestra.run.smithi038.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-03-26T06:24:55.321 INFO:teuthology.orchestra.run.smithi038.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-26T06:24:55.346 INFO:teuthology.orchestra.run.smithi038.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2024-03-26T06:24:55.363 INFO:teuthology.orchestra.run.smithi038.stdout:(118/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 398 kB/s | 43 kB 00:00 2024-03-26T06:24:55.397 INFO:teuthology.orchestra.run.smithi038.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 198 kB/s | 15 kB 00:00 2024-03-26T06:24:55.422 INFO:teuthology.orchestra.run.smithi038.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-26T06:24:55.447 INFO:teuthology.orchestra.run.smithi038.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-26T06:24:55.464 INFO:teuthology.orchestra.run.smithi038.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 843 kB/s | 35 kB 00:00 2024-03-26T06:24:55.490 INFO:teuthology.orchestra.run.smithi038.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-03-26T06:24:55.515 INFO:teuthology.orchestra.run.smithi038.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-03-26T06:24:55.540 INFO:teuthology.orchestra.run.smithi038.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-03-26T06:24:55.557 INFO:teuthology.orchestra.run.smithi038.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 256 kB/s | 11 kB 00:00 2024-03-26T06:24:55.574 INFO:teuthology.orchestra.run.smithi038.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 526 kB/s | 18 kB 00:00 2024-03-26T06:24:55.591 INFO:teuthology.orchestra.run.smithi038.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 692 kB/s | 23 kB 00:00 2024-03-26T06:24:55.599 INFO:teuthology.orchestra.run.smithi121.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 115 kB/s | 248 kB 00:02 2024-03-26T06:24:55.608 INFO:teuthology.orchestra.run.smithi038.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 583 kB/s | 19 kB 00:00 2024-03-26T06:24:55.625 INFO:teuthology.orchestra.run.smithi038.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 588 kB/s | 20 kB 00:00 2024-03-26T06:24:55.683 INFO:teuthology.orchestra.run.smithi038.stdout:(131/174): python3-grpcio-1.46.7-10.el9.x86_64. 9.3 MB/s | 2.0 MB 00:00 2024-03-26T06:24:55.700 INFO:teuthology.orchestra.run.smithi038.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 615 kB/s | 46 kB 00:00 2024-03-26T06:24:55.725 INFO:teuthology.orchestra.run.smithi038.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-03-26T06:24:55.749 INFO:teuthology.orchestra.run.smithi121.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 476 kB/s | 71 kB 00:00 2024-03-26T06:24:55.759 INFO:teuthology.orchestra.run.smithi038.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-03-26T06:24:55.776 INFO:teuthology.orchestra.run.smithi038.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 764 kB/s | 58 kB 00:00 2024-03-26T06:24:55.793 INFO:teuthology.orchestra.run.smithi038.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 491 kB/s | 16 kB 00:00 2024-03-26T06:24:55.818 INFO:teuthology.orchestra.run.smithi038.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 2.1 MB/s | 90 kB 00:00 2024-03-26T06:24:55.835 INFO:teuthology.orchestra.run.smithi038.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 673 kB/s | 28 kB 00:00 2024-03-26T06:24:55.861 INFO:teuthology.orchestra.run.smithi038.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 730 kB/s | 31 kB 00:00 2024-03-26T06:24:55.886 INFO:teuthology.orchestra.run.smithi038.stdout:(140/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-03-26T06:24:55.903 INFO:teuthology.orchestra.run.smithi038.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-03-26T06:24:55.920 INFO:teuthology.orchestra.run.smithi038.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 1.0 MB/s | 36 kB 00:00 2024-03-26T06:24:55.924 INFO:teuthology.orchestra.run.smithi121.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 304 kB/s | 1.8 MB 00:06 2024-03-26T06:24:55.945 INFO:teuthology.orchestra.run.smithi038.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-03-26T06:24:55.962 INFO:teuthology.orchestra.run.smithi038.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 867 kB/s | 51 kB 00:00 2024-03-26T06:24:55.987 INFO:teuthology.orchestra.run.smithi038.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-03-26T06:24:56.013 INFO:teuthology.orchestra.run.smithi038.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-26T06:24:56.038 INFO:teuthology.orchestra.run.smithi038.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-03-26T06:24:56.063 INFO:teuthology.orchestra.run.smithi038.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-26T06:24:56.097 INFO:teuthology.orchestra.run.smithi038.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-03-26T06:24:56.164 INFO:teuthology.orchestra.run.smithi038.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-03-26T06:24:56.231 INFO:teuthology.orchestra.run.smithi038.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 4.7 MB/s | 321 kB 00:00 2024-03-26T06:24:56.389 INFO:teuthology.orchestra.run.smithi038.stdout:(152/174): NetworkManager-1.47.2-1.el9.x86_64.r 15 MB/s | 2.3 MB 00:00 2024-03-26T06:24:56.448 INFO:teuthology.orchestra.run.smithi038.stdout:(153/174): NetworkManager-libnm-1.47.2-1.el9.x8 32 MB/s | 1.8 MB 00:00 2024-03-26T06:24:56.490 INFO:teuthology.orchestra.run.smithi038.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-03-26T06:24:56.533 INFO:teuthology.orchestra.run.smithi038.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 5.9 MB/s | 248 kB 00:00 2024-03-26T06:24:56.574 INFO:teuthology.orchestra.run.smithi038.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 71 kB 00:00 2024-03-26T06:24:56.616 INFO:teuthology.orchestra.run.smithi038.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 878 kB/s | 36 kB 00:00 2024-03-26T06:24:56.658 INFO:teuthology.orchestra.run.smithi038.stdout:(158/174): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 192 kB 00:00 2024-03-26T06:24:56.671 INFO:teuthology.orchestra.run.smithi170.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-26T06:24:56.699 INFO:teuthology.orchestra.run.smithi038.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 256 kB 00:00 2024-03-26T06:24:56.734 INFO:teuthology.orchestra.run.smithi038.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 694 kB/s | 23 kB 00:00 2024-03-26T06:24:56.775 INFO:teuthology.orchestra.run.smithi038.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.5 MB/s | 108 kB 00:00 2024-03-26T06:24:56.834 INFO:teuthology.orchestra.run.smithi038.stdout:(162/174): librados2-19.0.0-1964.g7e98f5f6.el9. 4.4 MB/s | 3.4 MB 00:00 2024-03-26T06:24:56.859 INFO:teuthology.orchestra.run.smithi038.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 155 kB 00:00 2024-03-26T06:24:56.892 INFO:teuthology.orchestra.run.smithi038.stdout:(164/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.9 MB/s | 64 kB 00:00 2024-03-26T06:24:56.926 INFO:teuthology.orchestra.run.smithi038.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 136 kB 00:00 2024-03-26T06:24:56.951 INFO:teuthology.orchestra.run.smithi038.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 488 kB/s | 29 kB 00:00 2024-03-26T06:24:57.018 INFO:teuthology.orchestra.run.smithi038.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 33 MB/s | 2.2 MB 00:00 2024-03-26T06:24:57.069 INFO:teuthology.orchestra.run.smithi038.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 8.5 MB/s | 1.2 MB 00:00 2024-03-26T06:24:57.136 INFO:teuthology.orchestra.run.smithi038.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 19 MB/s | 2.3 MB 00:00 2024-03-26T06:24:57.161 INFO:teuthology.orchestra.run.smithi038.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 4.9 MB/s | 465 kB 00:00 2024-03-26T06:24:57.186 INFO:teuthology.orchestra.run.smithi038.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 622 kB/s | 31 kB 00:00 2024-03-26T06:24:57.220 INFO:teuthology.orchestra.run.smithi038.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-26T06:24:57.245 INFO:teuthology.orchestra.run.smithi038.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 74 kB 00:00 2024-03-26T06:24:57.808 INFO:teuthology.orchestra.run.smithi121.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 18 kB/s | 36 kB 00:02 2024-03-26T06:24:57.846 INFO:teuthology.orchestra.run.smithi038.stdout:(174/174): librbd1-19.0.0-1964.g7e98f5f6.el9.x8 1.8 MB/s | 3.1 MB 00:01 2024-03-26T06:24:57.849 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-26T06:24:57.850 INFO:teuthology.orchestra.run.smithi038.stdout:Total 15 MB/s | 223 MB 00:14 2024-03-26T06:24:57.933 INFO:teuthology.orchestra.run.smithi121.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 96 kB/s | 192 kB 00:02 2024-03-26T06:24:58.427 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2024-03-26T06:24:58.724 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2024-03-26T06:24:58.724 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2024-03-26T06:24:59.891 INFO:teuthology.orchestra.run.smithi121.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 12 kB/s | 23 kB 00:01 2024-03-26T06:25:00.029 INFO:teuthology.orchestra.run.smithi121.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 115 kB/s | 256 kB 00:02 2024-03-26T06:25:00.847 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2024-03-26T06:25:00.847 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2024-03-26T06:25:01.567 INFO:teuthology.orchestra.run.smithi170.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-26T06:25:02.108 INFO:teuthology.orchestra.run.smithi121.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 49 kB/s | 108 kB 00:02 2024-03-26T06:25:02.158 INFO:teuthology.orchestra.run.smithi121.stdout:(163/174): NetworkManager-1.47.2-1.el9.x86_64.r 185 kB/s | 2.3 MB 00:12 2024-03-26T06:25:02.183 INFO:teuthology.orchestra.run.smithi121.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 72 kB/s | 155 kB 00:02 2024-03-26T06:25:02.284 INFO:teuthology.orchestra.run.smithi121.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 287 kB/s | 29 kB 00:00 2024-03-26T06:25:02.309 INFO:teuthology.orchestra.run.smithi121.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 677 kB/s | 136 kB 00:00 2024-03-26T06:25:02.334 INFO:teuthology.orchestra.run.smithi121.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 424 kB/s | 64 kB 00:00 2024-03-26T06:25:03.947 INFO:teuthology.orchestra.run.smithi170.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-26T06:25:03.947 INFO:teuthology.orchestra.run.smithi170.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-26T06:25:03.959 INFO:teuthology.orchestra.run.smithi170.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-26T06:25:03.960 INFO:teuthology.orchestra.run.smithi170.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-26T06:25:03.961 INFO:teuthology.orchestra.run.smithi170.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-26T06:25:03.962 INFO:teuthology.orchestra.run.smithi170.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-26T06:25:03.963 INFO:teuthology.orchestra.run.smithi170.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-26T06:25:03.966 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-26T06:25:03.967 INFO:teuthology.orchestra.run.smithi170.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-26T06:25:04.045 INFO:teuthology.orchestra.run.smithi170.stdout:Dependencies resolved. 2024-03-26T06:25:04.060 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-26T06:25:04.060 INFO:teuthology.orchestra.run.smithi170.stdout: Package Arch Version Repository Size 2024-03-26T06:25:04.060 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-26T06:25:04.060 INFO:teuthology.orchestra.run.smithi170.stdout:Installing: 2024-03-26T06:25:04.060 INFO:teuthology.orchestra.run.smithi170.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-26T06:25:04.060 INFO:teuthology.orchestra.run.smithi170.stdout: ceph x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 6.6 k 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 5.4 M 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 889 k 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 145 k 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 1.5 M 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 139 k 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 1.8 M 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 7.4 M 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 49 k 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 9.8 M 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 47 M 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 264 k 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 749 k 2024-03-26T06:25:04.061 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 219 k 2024-03-26T06:25:04.062 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-shell x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 27 k 2024-03-26T06:25:04.062 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 18 k 2024-03-26T06:25:04.062 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-26T06:25:04.062 INFO:teuthology.orchestra.run.smithi170.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-26T06:25:04.062 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-26T06:25:04.062 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 31 k 2024-03-26T06:25:04.062 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 745 k 2024-03-26T06:25:04.062 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 127 k 2024-03-26T06:25:04.062 INFO:teuthology.orchestra.run.smithi170.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-26T06:25:04.062 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-26T06:25:04.062 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-26T06:25:04.063 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-03-26T06:25:04.063 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-03-26T06:25:04.063 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-03-26T06:25:04.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 162 k 2024-03-26T06:25:04.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 321 k 2024-03-26T06:25:04.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 298 k 2024-03-26T06:25:04.063 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 99 k 2024-03-26T06:25:04.063 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 86 k 2024-03-26T06:25:04.063 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 3.1 M 2024-03-26T06:25:04.063 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 170 k 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrading: 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-26T06:25:04.064 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: librados2 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 3.4 M 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 3.1 M 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-26T06:25:04.065 INFO:teuthology.orchestra.run.smithi170.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout:Installing dependencies: 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 20 M 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 34 k 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 2.1 M 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 248 k 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 4.7 M 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 17 M 2024-03-26T06:25:04.066 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1964.g7e98f5f6.el9 ceph-noarch 16 k 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 25 k 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 163 k 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T06:25:04.067 INFO:teuthology.orchestra.run.smithi170.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 503 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2 x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 5.1 M 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T06:25:04.068 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 45 k 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common x86_64 2:19.0.0-1964.g7e98f5f6.el9 ceph 131 k 2024-03-26T06:25:04.069 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T06:25:04.070 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T06:25:04.071 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-26T06:25:04.072 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T06:25:04.073 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout:Installing weak dependencies: 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T06:25:04.074 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:25:04.075 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction Summary 2024-03-26T06:25:04.075 INFO:teuthology.orchestra.run.smithi170.stdout:============================================================================================ 2024-03-26T06:25:04.075 INFO:teuthology.orchestra.run.smithi170.stdout:Install 150 Packages 2024-03-26T06:25:04.075 INFO:teuthology.orchestra.run.smithi170.stdout:Upgrade 24 Packages 2024-03-26T06:25:04.075 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:25:04.075 INFO:teuthology.orchestra.run.smithi170.stdout:Total download size: 223 M 2024-03-26T06:25:04.076 INFO:teuthology.orchestra.run.smithi170.stdout:Downloading Packages: 2024-03-26T06:25:04.734 INFO:teuthology.orchestra.run.smithi121.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 506 kB/s | 1.2 MB 00:02 2024-03-26T06:25:04.827 INFO:teuthology.orchestra.run.smithi170.stdout:(1/174): ceph-19.0.0-1964.g7e98f5f6.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-03-26T06:25:05.020 INFO:teuthology.orchestra.run.smithi121.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 814 kB/s | 2.2 MB 00:02 2024-03-26T06:25:05.027 INFO:teuthology.orchestra.run.smithi170.stdout:(2/174): ceph-fuse-19.0.0-1964.g7e98f5f6.el9.x8 4.3 MB/s | 889 kB 00:00 2024-03-26T06:25:05.045 INFO:teuthology.orchestra.run.smithi121.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 1.5 MB/s | 465 kB 00:00 2024-03-26T06:25:05.103 INFO:teuthology.orchestra.run.smithi121.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 373 kB/s | 31 kB 00:00 2024-03-26T06:25:05.110 INFO:teuthology.orchestra.run.smithi170.stdout:(3/174): ceph-immutable-object-cache-19.0.0-196 1.7 MB/s | 145 kB 00:00 2024-03-26T06:25:05.187 INFO:teuthology.orchestra.run.smithi121.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 892 kB/s | 74 kB 00:00 2024-03-26T06:25:05.346 INFO:teuthology.orchestra.run.smithi121.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 776 kB/s | 2.3 MB 00:03 2024-03-26T06:25:05.352 INFO:teuthology.orchestra.run.smithi170.stdout:(4/174): ceph-base-19.0.0-1964.g7e98f5f6.el9.x8 7.7 MB/s | 5.4 MB 00:00 2024-03-26T06:25:05.428 INFO:teuthology.orchestra.run.smithi170.stdout:(5/174): ceph-mds-19.0.0-1964.g7e98f5f6.el9.x86 6.7 MB/s | 2.1 MB 00:00 2024-03-26T06:25:05.511 INFO:teuthology.orchestra.run.smithi170.stdout:(6/174): ceph-mgr-19.0.0-1964.g7e98f5f6.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-03-26T06:25:05.579 INFO:teuthology.orchestra.run.smithi121.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 2.0 MB/s | 1.1 MB 00:00 2024-03-26T06:25:05.583 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-03-26T06:25:05.584 INFO:teuthology.orchestra.run.smithi121.stdout:Total 7.6 MB/s | 223 MB 00:29 2024-03-26T06:25:06.144 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-03-26T06:25:06.186 INFO:teuthology.orchestra.run.smithi170.stdout:(7/174): ceph-mon-19.0.0-1964.g7e98f5f6.el9.x86 6.2 MB/s | 4.7 MB 00:00 2024-03-26T06:25:06.416 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-03-26T06:25:06.417 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-03-26T06:25:06.887 INFO:teuthology.orchestra.run.smithi170.stdout:(8/174): ceph-common-19.0.0-1964.g7e98f5f6.el9. 8.9 MB/s | 20 MB 00:02 2024-03-26T06:25:06.979 INFO:teuthology.orchestra.run.smithi170.stdout:(9/174): ceph-selinux-19.0.0-1964.g7e98f5f6.el9 275 kB/s | 25 kB 00:00 2024-03-26T06:25:07.387 INFO:teuthology.orchestra.run.smithi170.stdout:(10/174): ceph-osd-19.0.0-1964.g7e98f5f6.el9.x8 9.0 MB/s | 17 MB 00:01 2024-03-26T06:25:07.446 INFO:teuthology.orchestra.run.smithi170.stdout:(11/174): cephfs-mirror-19.0.0-1964.g7e98f5f6.e 3.7 MB/s | 219 kB 00:00 2024-03-26T06:25:07.513 INFO:teuthology.orchestra.run.smithi170.stdout:(12/174): cephfs-shell-19.0.0-1964.g7e98f5f6.el 405 kB/s | 27 kB 00:00 2024-03-26T06:25:07.563 INFO:teuthology.orchestra.run.smithi170.stdout:(13/174): libcephfs-devel-19.0.0-1964.g7e98f5f6 627 kB/s | 31 kB 00:00 2024-03-26T06:25:07.680 INFO:teuthology.orchestra.run.smithi170.stdout:(14/174): libcephfs2-19.0.0-1964.g7e98f5f6.el9. 6.2 MB/s | 745 kB 00:00 2024-03-26T06:25:07.747 INFO:teuthology.orchestra.run.smithi170.stdout:(15/174): libcephsqlite-19.0.0-1964.g7e98f5f6.e 2.4 MB/s | 163 kB 00:00 2024-03-26T06:25:07.856 INFO:teuthology.orchestra.run.smithi170.stdout:(16/174): ceph-radosgw-19.0.0-1964.g7e98f5f6.el 5.9 MB/s | 9.8 MB 00:01 2024-03-26T06:25:07.881 INFO:teuthology.orchestra.run.smithi170.stdout:(17/174): librados-devel-19.0.0-1964.g7e98f5f6. 948 kB/s | 127 kB 00:00 2024-03-26T06:25:07.956 INFO:teuthology.orchestra.run.smithi170.stdout:(18/174): libradosstriper1-19.0.0-1964.g7e98f5f 4.9 MB/s | 503 kB 00:00 2024-03-26T06:25:08.023 INFO:teuthology.orchestra.run.smithi170.stdout:(19/174): python3-ceph-argparse-19.0.0-1964.g7e 678 kB/s | 45 kB 00:00 2024-03-26T06:25:08.030 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2024-03-26T06:25:08.090 INFO:teuthology.orchestra.run.smithi170.stdout:(20/174): python3-ceph-common-19.0.0-1964.g7e98 1.9 MB/s | 131 kB 00:00 2024-03-26T06:25:08.125 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-26T06:25:08.157 INFO:teuthology.orchestra.run.smithi170.stdout:(21/174): python3-cephfs-19.0.0-1964.g7e98f5f6. 2.4 MB/s | 162 kB 00:00 2024-03-26T06:25:08.160 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-26T06:25:08.198 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-26T06:25:08.206 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-03-26T06:25:08.206 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-03-26T06:25:08.207 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-26T06:25:08.263 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-26T06:25:08.274 INFO:teuthology.orchestra.run.smithi170.stdout:(22/174): python3-rados-19.0.0-1964.g7e98f5f6.e 2.7 MB/s | 321 kB 00:00 2024-03-26T06:25:08.319 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-26T06:25:08.349 INFO:teuthology.orchestra.run.smithi170.stdout:(23/174): python3-rbd-19.0.0-1964.g7e98f5f6.el9 3.9 MB/s | 298 kB 00:00 2024-03-26T06:25:08.360 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-26T06:25:08.385 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-26T06:25:08.439 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-26T06:25:08.533 INFO:teuthology.orchestra.run.smithi170.stdout:(24/174): librgw2-19.0.0-1964.g7e98f5f6.el9.x86 7.9 MB/s | 5.1 MB 00:00 2024-03-26T06:25:08.558 INFO:teuthology.orchestra.run.smithi170.stdout:(25/174): python3-rgw-19.0.0-1964.g7e98f5f6.el9 477 kB/s | 99 kB 00:00 2024-03-26T06:25:08.600 INFO:teuthology.orchestra.run.smithi170.stdout:(26/174): rbd-fuse-19.0.0-1964.g7e98f5f6.el9.x8 1.3 MB/s | 86 kB 00:00 2024-03-26T06:25:08.667 INFO:teuthology.orchestra.run.smithi170.stdout:(27/174): rbd-nbd-19.0.0-1964.g7e98f5f6.el9.x86 2.5 MB/s | 170 kB 00:00 2024-03-26T06:25:08.726 INFO:teuthology.orchestra.run.smithi170.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1964.g 582 kB/s | 34 kB 00:00 2024-03-26T06:25:08.782 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-26T06:25:08.793 INFO:teuthology.orchestra.run.smithi170.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1964.g7e98f5f 2.0 MB/s | 139 kB 00:00 2024-03-26T06:25:08.836 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 10/198 2024-03-26T06:25:08.968 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 10/198 2024-03-26T06:25:08.995 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 11/198 2024-03-26T06:25:09.027 INFO:teuthology.orchestra.run.smithi170.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1964.g7e98f 7.7 MB/s | 1.8 MB 00:00 2024-03-26T06:25:09.085 INFO:teuthology.orchestra.run.smithi170.stdout:(31/174): rbd-mirror-19.0.0-1964.g7e98f5f6.el9. 5.9 MB/s | 3.1 MB 00:00 2024-03-26T06:25:09.088 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 11/198 2024-03-26T06:25:09.117 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:19.0.0-1964.g7e98f5f6.el9.x86_64 12/198 2024-03-26T06:25:09.153 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-26T06:25:09.177 INFO:teuthology.orchestra.run.smithi170.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1964.g7e 2.6 MB/s | 248 kB 00:00 2024-03-26T06:25:09.252 INFO:teuthology.orchestra.run.smithi170.stdout:(33/174): ceph-mgr-rook-19.0.0-1964.g7e98f5f6.e 658 kB/s | 49 kB 00:00 2024-03-26T06:25:09.311 INFO:teuthology.orchestra.run.smithi170.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1964.g7 283 kB/s | 16 kB 00:00 2024-03-26T06:25:09.360 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-26T06:25:09.386 INFO:teuthology.orchestra.run.smithi170.stdout:(35/174): ceph-volume-19.0.0-1964.g7e98f5f6.el9 3.4 MB/s | 264 kB 00:00 2024-03-26T06:25:09.415 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-26T06:25:09.451 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 16/198 2024-03-26T06:25:09.557 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 16/198 2024-03-26T06:25:09.561 INFO:teuthology.orchestra.run.smithi170.stdout:(36/174): cephadm-19.0.0-1964.g7e98f5f6.el9.noa 4.2 MB/s | 749 kB 00:00 2024-03-26T06:25:09.577 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86 17/198 2024-03-26T06:25:09.633 INFO:teuthology.orchestra.run.smithi170.stdout:(37/174): cephfs-top-19.0.0-1964.g7e98f5f6.el9. 256 kB/s | 18 kB 00:00 2024-03-26T06:25:09.642 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86 17/198 2024-03-26T06:25:09.673 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-26T06:25:09.696 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-26T06:25:09.711 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-26T06:25:09.817 INFO:teuthology.orchestra.run.smithi170.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 9.4 MB/s | 7.4 MB 00:00 2024-03-26T06:25:09.854 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-26T06:25:09.886 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-26T06:25:09.934 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-26T06:25:10.050 INFO:teuthology.orchestra.run.smithi170.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 1.6 MB/s | 662 kB 00:00 2024-03-26T06:25:10.115 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-26T06:25:10.126 INFO:teuthology.orchestra.run.smithi170.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 2.9 MB/s | 920 kB 00:00 2024-03-26T06:25:10.151 INFO:teuthology.orchestra.run.smithi170.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.0 MB/s | 106 kB 00:00 2024-03-26T06:25:10.202 INFO:teuthology.orchestra.run.smithi170.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.0 MB/s | 307 kB 00:00 2024-03-26T06:25:10.227 INFO:teuthology.orchestra.run.smithi170.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 420 kB/s | 32 kB 00:00 2024-03-26T06:25:10.255 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-26T06:25:10.315 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-26T06:25:10.319 INFO:teuthology.orchestra.run.smithi170.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 187 kB/s | 17 kB 00:00 2024-03-26T06:25:10.404 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-26T06:25:10.455 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-26T06:25:10.666 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-26T06:25:10.777 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-26T06:25:10.778 INFO:teuthology.orchestra.run.smithi170.stdout:(45/174): ceph-test-19.0.0-1964.g7e98f5f6.el9.x 12 MB/s | 47 MB 00:03 2024-03-26T06:25:10.845 INFO:teuthology.orchestra.run.smithi170.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.6 MB/s | 3.0 MB 00:00 2024-03-26T06:25:10.870 INFO:teuthology.orchestra.run.smithi170.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 18 kB/s | 10 kB 00:00 2024-03-26T06:25:10.872 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-26T06:25:10.895 INFO:teuthology.orchestra.run.smithi170.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 159 kB 00:00 2024-03-26T06:25:10.915 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-26T06:25:10.937 INFO:teuthology.orchestra.run.smithi170.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 666 kB/s | 44 kB 00:00 2024-03-26T06:25:10.942 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-26T06:25:10.962 INFO:teuthology.orchestra.run.smithi170.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 661 kB 00:00 2024-03-26T06:25:10.986 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-26T06:25:10.996 INFO:teuthology.orchestra.run.smithi170.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 5.6 MB/s | 578 kB 00:00 2024-03-26T06:25:11.021 INFO:teuthology.orchestra.run.smithi170.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 561 kB/s | 46 kB 00:00 2024-03-26T06:25:11.046 INFO:teuthology.orchestra.run.smithi170.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-26T06:25:11.063 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-26T06:25:11.072 INFO:teuthology.orchestra.run.smithi170.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-03-26T06:25:11.097 INFO:teuthology.orchestra.run.smithi170.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-26T06:25:11.122 INFO:teuthology.orchestra.run.smithi170.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 456 kB/s | 34 kB 00:00 2024-03-26T06:25:11.123 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-26T06:25:11.175 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-26T06:25:11.189 INFO:teuthology.orchestra.run.smithi170.stdout:(57/174): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-03-26T06:25:11.244 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-26T06:25:11.285 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-26T06:25:11.289 INFO:teuthology.orchestra.run.smithi170.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 25 MB/s | 4.7 MB 00:00 2024-03-26T06:25:11.315 INFO:teuthology.orchestra.run.smithi170.stdout:(59/174): python3-attrs-20.3.0-7.el9.noarch.rpm 408 kB/s | 78 kB 00:00 2024-03-26T06:25:11.320 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-26T06:25:11.353 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-26T06:25:11.380 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-26T06:25:11.440 INFO:teuthology.orchestra.run.smithi170.stdout:(60/174): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-26T06:25:11.453 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-26T06:25:11.457 INFO:teuthology.orchestra.run.smithi170.stdout:(61/174): python3-jsonpatch-1.21-16.el9.noarch. 149 kB/s | 24 kB 00:00 2024-03-26T06:25:11.474 INFO:teuthology.orchestra.run.smithi170.stdout:(62/174): python3-jsonpointer-2.0-4.el9.noarch. 116 kB/s | 18 kB 00:00 2024-03-26T06:25:11.507 INFO:teuthology.orchestra.run.smithi170.stdout:(63/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-26T06:25:11.510 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:19.0.0-1964.g7e98f5f6.el9. 39/198 2024-03-26T06:25:11.533 INFO:teuthology.orchestra.run.smithi170.stdout:(64/174): python3-markupsafe-1.1.1-12.el9.x86_6 541 kB/s | 32 kB 00:00 2024-03-26T06:25:11.554 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:19.0.0-1964.g7e98f5f6.el 40/198 2024-03-26T06:25:11.582 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:19.0.0-1964.g7e98f5f6.el9.x86_6 41/198 2024-03-26T06:25:11.650 INFO:teuthology.orchestra.run.smithi170.stdout:(65/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 27 MB/s | 4.8 MB 00:00 2024-03-26T06:25:11.655 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-26T06:25:11.675 INFO:teuthology.orchestra.run.smithi170.stdout:(66/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-26T06:25:11.701 INFO:teuthology.orchestra.run.smithi170.stdout:(67/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.0 MB/s | 174 kB 00:00 2024-03-26T06:25:11.725 INFO:teuthology.orchestra.run.smithi170.stdout:(68/174): python3-packaging-20.9-5.el9.noarch.r 923 kB/s | 69 kB 00:00 2024-03-26T06:25:11.751 INFO:teuthology.orchestra.run.smithi170.stdout:(69/174): python3-prettytable-0.7.2-27.el9.noar 823 kB/s | 41 kB 00:00 2024-03-26T06:25:11.784 INFO:teuthology.orchestra.run.smithi170.stdout:(70/174): python3-protobuf-3.14.0-13.el9.noarch 2.8 MB/s | 238 kB 00:00 2024-03-26T06:25:11.809 INFO:teuthology.orchestra.run.smithi170.stdout:(71/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-26T06:25:11.835 INFO:teuthology.orchestra.run.smithi170.stdout:(72/174): python3-pytz-2021.1-5.el9.noarch.rpm 952 kB/s | 47 kB 00:00 2024-03-26T06:25:11.860 INFO:teuthology.orchestra.run.smithi170.stdout:(73/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-26T06:25:11.885 INFO:teuthology.orchestra.run.smithi170.stdout:(74/174): python3-requests-oauthlib-1.3.0-12.el 568 kB/s | 43 kB 00:00 2024-03-26T06:25:11.889 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-26T06:25:12.057 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-26T06:25:12.136 INFO:teuthology.orchestra.run.smithi170.stdout:(75/174): python3-toml-0.10.2-6.el9.noarch.rpm 156 kB/s | 39 kB 00:00 2024-03-26T06:25:12.161 INFO:teuthology.orchestra.run.smithi170.stdout:(76/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 151 kB/s | 41 kB 00:00 2024-03-26T06:25:12.195 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-26T06:25:12.250 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-26T06:25:12.370 INFO:teuthology.orchestra.run.smithi170.stdout:(77/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-26T06:25:12.403 INFO:teuthology.orchestra.run.smithi170.stdout:(78/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 300 kB 00:00 2024-03-26T06:25:12.428 INFO:teuthology.orchestra.run.smithi170.stdout:(79/174): uuid-1.6.2-55.el9.x86_64.rpm 213 kB/s | 56 kB 00:00 2024-03-26T06:25:12.454 INFO:teuthology.orchestra.run.smithi170.stdout:(80/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 758 kB/s | 63 kB 00:00 2024-03-26T06:25:12.479 INFO:teuthology.orchestra.run.smithi170.stdout:(81/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-26T06:25:12.504 INFO:teuthology.orchestra.run.smithi170.stdout:(82/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-26T06:25:12.529 INFO:teuthology.orchestra.run.smithi170.stdout:(83/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-26T06:25:12.555 INFO:teuthology.orchestra.run.smithi170.stdout:(84/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-26T06:25:12.588 INFO:teuthology.orchestra.run.smithi170.stdout:(85/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-26T06:25:12.613 INFO:teuthology.orchestra.run.smithi170.stdout:(86/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-26T06:25:12.639 INFO:teuthology.orchestra.run.smithi170.stdout:(87/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-26T06:25:12.804 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-26T06:25:12.880 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-26T06:25:12.969 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-26T06:25:12.989 INFO:teuthology.orchestra.run.smithi170.stdout:(88/174): jq-1.6-16.el9.x86_64.rpm 499 kB/s | 187 kB 00:00 2024-03-26T06:25:13.022 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-26T06:25:13.033 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-26T06:25:13.056 INFO:teuthology.orchestra.run.smithi170.stdout:(89/174): libquadmath-11.4.1-3.el9.x86_64.rpm 468 kB/s | 195 kB 00:00 2024-03-26T06:25:13.072 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-26T06:25:13.120 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-26T06:25:13.149 INFO:teuthology.orchestra.run.smithi170.stdout:(90/174): python3-requests-2.25.1-8.el9.noarch. 780 kB/s | 125 kB 00:00 2024-03-26T06:25:13.158 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_6 53/198 2024-03-26T06:25:13.188 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-26T06:25:13.241 INFO:teuthology.orchestra.run.smithi170.stdout:(91/174): elfutils-libelf-devel-0.190-2.el9.x86 116 kB/s | 21 kB 00:00 2024-03-26T06:25:13.275 INFO:teuthology.orchestra.run.smithi170.stdout:(92/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 203 kB/s | 25 kB 00:00 2024-03-26T06:25:13.469 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-26T06:25:13.483 INFO:teuthology.orchestra.run.smithi170.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 924 kB/s | 803 kB 00:00 2024-03-26T06:25:13.592 INFO:teuthology.orchestra.run.smithi170.stdout:(94/174): postgresql-13.14-1.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-03-26T06:25:13.650 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-26T06:25:13.651 INFO:teuthology.orchestra.run.smithi170.stdout:(95/174): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-03-26T06:25:13.765 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-26T06:25:13.809 INFO:teuthology.orchestra.run.smithi170.stdout:(96/174): postgresql-contrib-13.14-1.el9.x86_64 2.7 MB/s | 911 kB 00:00 2024-03-26T06:25:13.910 INFO:teuthology.orchestra.run.smithi170.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-26T06:25:13.953 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-26T06:25:14.010 INFO:teuthology.orchestra.run.smithi170.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-03-26T06:25:14.102 INFO:teuthology.orchestra.run.smithi170.stdout:(99/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 4.8 MB/s | 4.1 MB 00:00 2024-03-26T06:25:14.104 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-26T06:25:14.144 INFO:teuthology.orchestra.run.smithi170.stdout:(100/174): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-03-26T06:25:14.154 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-26T06:25:14.169 INFO:teuthology.orchestra.run.smithi170.stdout:(101/174): qatlib-service-24.02.0-1.el9.x86_64. 537 kB/s | 36 kB 00:00 2024-03-26T06:25:14.203 INFO:teuthology.orchestra.run.smithi170.stdout:(102/174): qatzip-libs-1.2.0-1.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-03-26T06:25:14.211 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-26T06:25:14.228 INFO:teuthology.orchestra.run.smithi170.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-26T06:25:14.284 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-26T06:25:14.295 INFO:teuthology.orchestra.run.smithi170.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-26T06:25:14.321 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-26T06:25:14.358 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-26T06:25:14.397 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-26T06:25:14.440 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-26T06:25:14.484 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-26T06:25:14.526 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-26T06:25:14.587 INFO:teuthology.orchestra.run.smithi170.stdout:(105/174): postgresql-server-13.14-1.el9.x86_64 6.2 MB/s | 5.8 MB 00:00 2024-03-26T06:25:14.670 INFO:teuthology.orchestra.run.smithi170.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-03-26T06:25:14.674 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-26T06:25:14.723 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-26T06:25:14.729 INFO:teuthology.orchestra.run.smithi170.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 332 kB/s | 19 kB 00:00 2024-03-26T06:25:14.754 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-26T06:25:14.763 INFO:teuthology.orchestra.run.smithi170.stdout:(108/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.6 MB/s | 863 kB 00:00 2024-03-26T06:25:14.795 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-26T06:25:14.821 INFO:teuthology.orchestra.run.smithi170.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 445 kB/s | 26 kB 00:00 2024-03-26T06:25:14.880 INFO:teuthology.orchestra.run.smithi170.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 831 kB/s | 49 kB 00:00 2024-03-26T06:25:14.894 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-26T06:25:14.905 INFO:teuthology.orchestra.run.smithi170.stdout:(111/174): gperftools-libs-2.9.1-2.el9.x86_64.r 971 kB/s | 309 kB 00:00 2024-03-26T06:25:14.927 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-26T06:25:14.939 INFO:teuthology.orchestra.run.smithi170.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-26T06:25:14.968 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-26T06:25:14.993 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-26T06:25:15.018 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-26T06:25:15.031 INFO:teuthology.orchestra.run.smithi170.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-03-26T06:25:15.065 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-03-26T06:25:15.206 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-26T06:25:15.207 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-26T06:25:15.208 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-26T06:25:15.224 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-26T06:25:15.247 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-26T06:25:15.248 INFO:teuthology.orchestra.run.smithi170.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 838 kB 00:00 2024-03-26T06:25:15.256 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-26T06:25:15.296 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-26T06:25:15.315 INFO:teuthology.orchestra.run.smithi170.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 652 kB/s | 43 kB 00:00 2024-03-26T06:25:15.341 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-26T06:25:15.348 INFO:teuthology.orchestra.run.smithi170.stdout:(116/174): python3-asyncssh-2.13.2-5.el9.noarch 1.7 MB/s | 548 kB 00:00 2024-03-26T06:25:15.350 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-26T06:25:15.373 INFO:teuthology.orchestra.run.smithi170.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 551 kB/s | 32 kB 00:00 2024-03-26T06:25:15.393 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-26T06:25:15.414 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-26T06:25:15.415 INFO:teuthology.orchestra.run.smithi170.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-03-26T06:25:15.443 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-26T06:25:15.452 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-26T06:25:15.474 INFO:teuthology.orchestra.run.smithi170.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-03-26T06:25:15.485 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-26T06:25:15.515 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-26T06:25:15.575 INFO:teuthology.orchestra.run.smithi170.stdout:(120/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-03-26T06:25:15.608 INFO:teuthology.orchestra.run.smithi170.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 1.8 MB/s | 358 kB 00:00 2024-03-26T06:25:15.642 INFO:teuthology.orchestra.run.smithi170.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 528 kB/s | 35 kB 00:00 2024-03-26T06:25:15.708 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-26T06:25:15.708 INFO:teuthology.orchestra.run.smithi170.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-26T06:25:15.756 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-26T06:25:15.806 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-26T06:25:15.809 INFO:teuthology.orchestra.run.smithi170.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-26T06:25:15.861 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 10/198 2024-03-26T06:25:15.867 INFO:teuthology.orchestra.run.smithi170.stdout:(125/174): libarrow-9.0.0-10.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-03-26T06:25:15.884 INFO:teuthology.orchestra.run.smithi170.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-03-26T06:25:15.900 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-26T06:25:15.927 INFO:teuthology.orchestra.run.smithi170.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-03-26T06:25:15.937 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-26T06:25:15.952 INFO:teuthology.orchestra.run.smithi170.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 346 kB/s | 23 kB 00:00 2024-03-26T06:25:15.969 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-26T06:25:15.982 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 10/198 2024-03-26T06:25:15.985 INFO:teuthology.orchestra.run.smithi170.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 332 kB/s | 19 kB 00:00 2024-03-26T06:25:15.999 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-26T06:25:16.002 INFO:teuthology.orchestra.run.smithi170.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-03-26T06:25:16.004 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 11/198 2024-03-26T06:25:16.028 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-26T06:25:16.053 INFO:teuthology.orchestra.run.smithi170.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 931 kB/s | 46 kB 00:00 2024-03-26T06:25:16.062 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-26T06:25:16.095 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 11/198 2024-03-26T06:25:16.110 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-26T06:25:16.111 INFO:teuthology.orchestra.run.smithi170.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-03-26T06:25:16.124 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:19.0.0-1964.g7e98f5f6.el9.x86_64 12/198 2024-03-26T06:25:16.141 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-26T06:25:16.153 INFO:teuthology.orchestra.run.smithi170.stdout:(133/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.0 MB/s | 2.0 MB 00:00 2024-03-26T06:25:16.166 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-26T06:25:16.171 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-26T06:25:16.187 INFO:teuthology.orchestra.run.smithi170.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-03-26T06:25:16.203 INFO:teuthology.orchestra.run.smithi170.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 626 kB/s | 58 kB 00:00 2024-03-26T06:25:16.246 INFO:teuthology.orchestra.run.smithi170.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-26T06:25:16.251 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-26T06:25:16.262 INFO:teuthology.orchestra.run.smithi170.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-03-26T06:25:16.288 INFO:teuthology.orchestra.run.smithi170.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-03-26T06:25:16.289 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-26T06:25:16.313 INFO:teuthology.orchestra.run.smithi170.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 420 kB/s | 28 kB 00:00 2024-03-26T06:25:16.323 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-26T06:25:16.333 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-26T06:25:16.338 INFO:teuthology.orchestra.run.smithi170.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-03-26T06:25:16.365 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-26T06:25:16.372 INFO:teuthology.orchestra.run.smithi170.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-26T06:25:16.389 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-26T06:25:16.397 INFO:teuthology.orchestra.run.smithi170.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 609 kB/s | 36 kB 00:00 2024-03-26T06:25:16.399 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-26T06:25:16.417 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 16/198 2024-03-26T06:25:16.422 INFO:teuthology.orchestra.run.smithi170.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 1.4 MB/s | 188 kB 00:00 2024-03-26T06:25:16.448 INFO:teuthology.orchestra.run.smithi170.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-03-26T06:25:16.473 INFO:teuthology.orchestra.run.smithi170.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-26T06:25:16.492 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-26T06:25:16.498 INFO:teuthology.orchestra.run.smithi170.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-26T06:25:16.521 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 16/198 2024-03-26T06:25:16.532 INFO:teuthology.orchestra.run.smithi170.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 339 kB/s | 20 kB 00:00 2024-03-26T06:25:16.543 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86 17/198 2024-03-26T06:25:16.565 INFO:teuthology.orchestra.run.smithi170.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-03-26T06:25:16.581 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-26T06:25:16.601 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86 17/198 2024-03-26T06:25:16.630 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-26T06:25:16.631 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-26T06:25:16.632 INFO:teuthology.orchestra.run.smithi170.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-26T06:25:16.653 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-26T06:25:16.661 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-26T06:25:16.669 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-26T06:25:16.709 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-26T06:25:16.775 INFO:teuthology.orchestra.run.smithi170.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-03-26T06:25:16.811 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-26T06:25:16.816 INFO:teuthology.orchestra.run.smithi170.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-26T06:25:16.861 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-26T06:25:16.864 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-26T06:25:16.892 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-26T06:25:16.924 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-26T06:25:17.008 INFO:teuthology.orchestra.run.smithi170.stdout:(152/174): librbd1-19.0.0-1964.g7e98f5f6.el9.x8 8.4 MB/s | 3.1 MB 00:00 2024-03-26T06:25:17.028 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-26T06:25:17.067 INFO:teuthology.orchestra.run.smithi170.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 9.2 MB/s | 2.3 MB 00:00 2024-03-26T06:25:17.126 INFO:teuthology.orchestra.run.smithi170.stdout:(154/174): librados2-19.0.0-1964.g7e98f5f6.el9. 6.0 MB/s | 3.4 MB 00:00 2024-03-26T06:25:17.128 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-26T06:25:17.151 INFO:teuthology.orchestra.run.smithi170.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 503 kB/s | 42 kB 00:00 2024-03-26T06:25:17.187 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-26T06:25:17.210 INFO:teuthology.orchestra.run.smithi170.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-26T06:25:17.260 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-26T06:25:17.277 INFO:teuthology.orchestra.run.smithi170.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 546 kB/s | 36 kB 00:00 2024-03-26T06:25:17.304 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-26T06:25:17.310 INFO:teuthology.orchestra.run.smithi170.stdout:(158/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.3 MB/s | 248 kB 00:00 2024-03-26T06:25:17.344 INFO:teuthology.orchestra.run.smithi170.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-26T06:25:17.403 INFO:teuthology.orchestra.run.smithi170.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 395 kB/s | 23 kB 00:00 2024-03-26T06:25:17.461 INFO:teuthology.orchestra.run.smithi170.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.7 MB/s | 256 kB 00:00 2024-03-26T06:25:17.503 INFO:teuthology.orchestra.run.smithi170.stdout:(162/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.7 MB/s | 1.8 MB 00:00 2024-03-26T06:25:17.517 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-26T06:25:17.528 INFO:teuthology.orchestra.run.smithi170.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 863 kB/s | 108 kB 00:00 2024-03-26T06:25:17.554 INFO:teuthology.orchestra.run.smithi170.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-26T06:25:17.587 INFO:teuthology.orchestra.run.smithi170.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-26T06:25:17.612 INFO:teuthology.orchestra.run.smithi170.stdout:(166/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-26T06:25:17.626 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-26T06:25:17.629 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-26T06:25:17.655 INFO:teuthology.orchestra.run.smithi170.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 286 kB/s | 29 kB 00:00 2024-03-26T06:25:17.676 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-26T06:25:17.710 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-26T06:25:17.712 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-26T06:25:17.743 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-26T06:25:17.759 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-26T06:25:17.763 INFO:teuthology.orchestra.run.smithi170.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 6.9 MB/s | 1.2 MB 00:00 2024-03-26T06:25:17.769 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-26T06:25:17.805 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-26T06:25:17.830 INFO:teuthology.orchestra.run.smithi170.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 6.8 MB/s | 465 kB 00:00 2024-03-26T06:25:17.851 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-26T06:25:17.879 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-26T06:25:17.886 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-26T06:25:17.889 INFO:teuthology.orchestra.run.smithi170.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 534 kB/s | 31 kB 00:00 2024-03-26T06:25:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-26T06:25:17.935 INFO:teuthology.orchestra.run.smithi170.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.8 MB/s | 2.2 MB 00:00 2024-03-26T06:25:17.972 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-26T06:25:17.998 INFO:teuthology.orchestra.run.smithi170.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.9 MB/s | 1.1 MB 00:00 2024-03-26T06:25:18.023 INFO:teuthology.orchestra.run.smithi170.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 808 kB/s | 74 kB 00:00 2024-03-26T06:25:18.035 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-26T06:25:18.063 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-26T06:25:18.079 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-26T06:25:18.100 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-26T06:25:18.123 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-26T06:25:18.155 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-26T06:25:18.180 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-26T06:25:18.182 INFO:teuthology.orchestra.run.smithi170.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.3 MB/s | 2.3 MB 00:00 2024-03-26T06:25:18.183 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-26T06:25:18.186 INFO:teuthology.orchestra.run.smithi170.stdout:-------------------------------------------------------------------------------- 2024-03-26T06:25:18.186 INFO:teuthology.orchestra.run.smithi170.stdout:Total 16 MB/s | 223 MB 00:14 2024-03-26T06:25:18.252 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-26T06:25:18.288 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:19.0.0-1964.g7e98f5f6.el9. 39/198 2024-03-26T06:25:18.331 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:19.0.0-1964.g7e98f5f6.el 40/198 2024-03-26T06:25:18.360 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:19.0.0-1964.g7e98f5f6.el9.x86_6 41/198 2024-03-26T06:25:18.429 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-26T06:25:18.491 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-26T06:25:18.565 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-26T06:25:18.706 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-26T06:25:18.737 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction check 2024-03-26T06:25:18.760 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-26T06:25:18.873 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-26T06:25:18.995 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-26T06:25:19.009 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction check succeeded. 2024-03-26T06:25:19.009 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction test 2024-03-26T06:25:19.277 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-26T06:25:19.356 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-26T06:25:19.425 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-26T06:25:19.484 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-26T06:25:19.494 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-26T06:25:19.524 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-26T06:25:19.565 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-26T06:25:19.603 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_6 53/198 2024-03-26T06:25:19.632 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-26T06:25:19.863 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-26T06:25:20.002 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-26T06:25:20.052 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-26T06:25:20.096 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-26T06:25:20.290 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-26T06:25:20.359 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-26T06:25:20.382 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-26T06:25:20.413 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-26T06:25:20.441 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-26T06:25:20.442 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-26T06:25:20.599 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-26T06:25:20.662 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-26T06:25:20.721 INFO:teuthology.orchestra.run.smithi170.stdout:Transaction test succeeded. 2024-03-26T06:25:20.722 INFO:teuthology.orchestra.run.smithi170.stdout:Running transaction 2024-03-26T06:25:20.737 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-26T06:25:20.769 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-26T06:25:20.770 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-26T06:25:20.804 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 124/198 2024-03-26T06:25:20.805 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-26T06:25:20.853 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-26T06:25:20.879 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 124/198 2024-03-26T06:25:20.891 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-26T06:25:20.909 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 125/198 2024-03-26T06:25:20.932 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-26T06:25:20.946 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-26T06:25:20.974 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-26T06:25:21.106 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-26T06:25:21.154 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-26T06:25:21.184 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-26T06:25:21.226 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-26T06:25:21.329 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-26T06:25:21.376 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-26T06:25:21.416 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-26T06:25:21.426 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-26T06:25:21.442 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-26T06:25:21.457 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 128/198 2024-03-26T06:25:21.467 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-26T06:25:21.528 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 128/198 2024-03-26T06:25:21.530 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-26T06:25:21.558 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 129/198 2024-03-26T06:25:21.659 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-26T06:25:21.775 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-26T06:25:21.817 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-26T06:25:21.864 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-26T06:25:21.921 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-26T06:25:22.102 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-26T06:25:22.146 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-26T06:25:22.275 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-26T06:25:22.311 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-26T06:25:22.343 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-26T06:25:22.374 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-26T06:25:22.402 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-26T06:25:22.436 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-26T06:25:22.475 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-26T06:25:22.507 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-26T06:25:22.537 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-26T06:25:22.612 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-26T06:25:22.647 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-26T06:25:22.681 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-26T06:25:22.722 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-26T06:25:22.757 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-26T06:25:22.844 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-26T06:25:22.926 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-26T06:25:22.979 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-26T06:25:23.011 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-26T06:25:23.058 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-26T06:25:23.154 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-26T06:25:23.203 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-26T06:25:23.850 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 130/198 2024-03-26T06:25:23.889 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 130/198 2024-03-26T06:25:23.925 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-26T06:25:23.967 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-26T06:25:24.001 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-26T06:25:24.049 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-26T06:25:24.135 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-26T06:25:24.170 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-26T06:25:24.348 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-26T06:25:24.378 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-26T06:25:24.452 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-26T06:25:24.457 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 130/198 2024-03-26T06:25:24.492 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 131/198 2024-03-26T06:25:24.589 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 131/198 2024-03-26T06:25:24.589 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T06:25:24.589 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T06:25:24.589 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:24.626 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 132/198 2024-03-26T06:25:25.074 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-26T06:25:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-26T06:25:26.378 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-26T06:25:26.467 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-26T06:25:26.499 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-26T06:25:26.530 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-26T06:25:26.557 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-26T06:25:26.868 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-26T06:25:26.896 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 124/198 2024-03-26T06:25:26.986 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 124/198 2024-03-26T06:25:27.017 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 125/198 2024-03-26T06:25:27.053 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-26T06:25:27.507 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-26T06:25:27.540 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 128/198 2024-03-26T06:25:27.602 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 128/198 2024-03-26T06:25:27.626 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 129/198 2024-03-26T06:25:27.867 INFO:teuthology.orchestra.run.smithi170.stdout: Preparing : 1/1 2024-03-26T06:25:27.977 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-26T06:25:28.011 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-26T06:25:28.042 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-26T06:25:28.051 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-26T06:25:28.091 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-26T06:25:28.145 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-26T06:25:28.179 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-26T06:25:28.213 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-26T06:25:28.254 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-26T06:25:28.575 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-26T06:25:28.639 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 10/198 2024-03-26T06:25:28.761 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 10/198 2024-03-26T06:25:28.790 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 11/198 2024-03-26T06:25:28.882 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 11/198 2024-03-26T06:25:28.913 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rados-2:19.0.0-1964.g7e98f5f6.el9.x86_64 12/198 2024-03-26T06:25:28.955 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 13/198 2024-03-26T06:25:29.129 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-26T06:25:29.186 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-13.14-1.el9.x86_64 15/198 2024-03-26T06:25:29.214 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 16/198 2024-03-26T06:25:29.316 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 16/198 2024-03-26T06:25:29.340 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86 17/198 2024-03-26T06:25:29.404 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86 17/198 2024-03-26T06:25:29.452 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-26T06:25:29.475 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-26T06:25:29.490 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 19/198 2024-03-26T06:25:29.683 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-26T06:25:29.714 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 20/198 2024-03-26T06:25:29.754 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-26T06:25:29.802 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 130/198 2024-03-26T06:25:29.847 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 130/198 2024-03-26T06:25:29.849 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-26T06:25:29.957 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-26T06:25:30.017 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-26T06:25:30.093 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-26T06:25:30.152 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-26T06:25:30.368 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-26T06:25:30.381 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 130/198 2024-03-26T06:25:30.417 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 131/198 2024-03-26T06:25:30.473 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-26T06:25:30.523 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 131/198 2024-03-26T06:25:30.523 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T06:25:30.524 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T06:25:30.524 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:25:30.551 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 132/198 2024-03-26T06:25:30.568 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-26T06:25:30.607 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-26T06:25:30.633 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-26T06:25:30.669 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-26T06:25:30.743 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-26T06:25:30.797 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-26T06:25:30.835 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-26T06:25:30.899 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-26T06:25:30.943 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-26T06:25:30.986 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-26T06:25:31.019 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-26T06:25:31.047 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-26T06:25:31.116 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-26T06:25:31.152 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-common-2:19.0.0-1964.g7e98f5f6.el9. 39/198 2024-03-26T06:25:31.195 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ceph-argparse-2:19.0.0-1964.g7e98f5f6.el 40/198 2024-03-26T06:25:31.224 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cephfs-2:19.0.0-1964.g7e98f5f6.el9.x86_6 41/198 2024-03-26T06:25:31.293 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-26T06:25:31.339 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-26T06:25:31.412 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-26T06:25:31.571 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-26T06:25:31.625 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-26T06:25:32.144 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-26T06:25:32.220 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-26T06:25:32.300 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-26T06:25:32.349 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-26T06:25:32.358 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-26T06:25:32.388 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 51/198 2024-03-26T06:25:32.429 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-26T06:25:32.467 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librados-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_6 53/198 2024-03-26T06:25:32.643 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-26T06:25:33.173 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-26T06:25:33.491 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-26T06:25:33.570 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-26T06:25:33.614 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-26T06:25:33.743 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-26T06:25:33.800 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-26T06:25:33.856 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-26T06:25:33.929 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-26T06:25:33.958 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-26T06:25:33.995 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-26T06:25:34.043 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-26T06:25:34.078 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-26T06:25:34.121 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-26T06:25:34.164 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-26T06:25:34.308 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-26T06:25:34.370 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-26T06:25:34.399 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-26T06:25:34.442 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-26T06:25:34.536 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-26T06:25:34.566 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-26T06:25:34.606 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-26T06:25:34.640 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-26T06:25:34.665 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-26T06:25:34.738 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-26T06:25:34.866 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-26T06:25:34.965 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-26T06:25:35.007 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-26T06:25:35.072 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-26T06:25:35.136 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-26T06:25:35.319 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-26T06:25:35.370 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-26T06:25:35.498 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-26T06:25:35.543 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-26T06:25:35.575 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-26T06:25:35.606 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-26T06:25:35.634 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-26T06:25:35.668 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-26T06:25:35.707 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-26T06:25:35.747 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-26T06:25:35.777 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-26T06:25:35.855 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-26T06:25:35.888 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-26T06:25:35.929 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-26T06:25:35.979 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-26T06:25:36.014 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-26T06:25:36.103 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-26T06:25:36.193 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-26T06:25:36.236 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-26T06:25:36.268 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-26T06:25:36.315 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-26T06:25:36.421 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-26T06:25:36.487 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-26T06:25:36.949 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 132/198 2024-03-26T06:25:36.949 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2024-03-26T06:25:36.949 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2024-03-26T06:25:36.949 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2024-03-26T06:25:36.949 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2024-03-26T06:25:36.949 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2024-03-26T06:25:36.949 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /root 2024-03-26T06:25:36.949 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2024-03-26T06:25:36.949 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:37.122 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 133/198 2024-03-26T06:25:37.136 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 133/198 2024-03-26T06:25:37.137 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:37.137 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T06:25:37.137 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:37.220 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-26T06:25:37.267 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-26T06:25:37.301 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-26T06:25:37.349 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-26T06:25:37.444 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-26T06:25:37.485 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-26T06:25:37.662 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-26T06:25:37.700 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-26T06:25:37.778 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-26T06:25:38.422 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-26T06:25:38.523 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-26T06:25:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 134/198 2024-03-26T06:25:38.699 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 134/198 2024-03-26T06:25:38.699 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:38.699 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T06:25:38.700 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T06:25:38.700 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T06:25:38.700 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:38.918 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 135/198 2024-03-26T06:25:38.946 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 135/198 2024-03-26T06:25:38.946 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:38.946 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T06:25:38.946 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T06:25:38.946 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T06:25:38.949 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:39.411 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 136/198 2024-03-26T06:25:39.433 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 136/198 2024-03-26T06:25:39.433 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:39.433 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T06:25:39.433 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T06:25:39.433 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T06:25:39.433 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:39.529 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-26T06:25:39.745 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-26T06:25:39.826 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-26T06:25:39.857 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-26T06:25:39.888 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-26T06:25:39.916 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-26T06:25:39.974 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 138/198 2024-03-26T06:25:40.002 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 138/198 2024-03-26T06:25:40.035 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1964.g7e98f5f6.e 139/198 2024-03-26T06:25:40.097 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1964.g7e98f5f6. 140/198 2024-03-26T06:25:40.125 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noa 141/198 2024-03-26T06:25:40.226 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-26T06:25:40.262 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 124/198 2024-03-26T06:25:40.352 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 124/198 2024-03-26T06:25:40.372 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noa 141/198 2024-03-26T06:25:40.384 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 125/198 2024-03-26T06:25:40.410 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.n 142/198 2024-03-26T06:25:40.420 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-26T06:25:40.874 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-26T06:25:40.899 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 128/198 2024-03-26T06:25:40.970 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 128/198 2024-03-26T06:25:40.992 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : python3-rgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 129/198 2024-03-26T06:25:41.424 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.n 142/198 2024-03-26T06:25:41.452 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e9 143/198 2024-03-26T06:25:41.560 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e9 143/198 2024-03-26T06:25:41.722 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1964.g7e98f5f6.el 144/198 2024-03-26T06:25:41.754 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 145/198 2024-03-26T06:25:41.776 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 145/198 2024-03-26T06:25:41.776 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:41.776 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T06:25:41.776 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T06:25:41.776 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T06:25:41.776 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:41.821 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 146/198 2024-03-26T06:25:41.831 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 146/198 2024-03-26T06:25:41.908 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:19.0.0-1964.g7e98f5f6.el9.x86_64 147/198 2024-03-26T06:25:42.350 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 132/198 2024-03-26T06:25:42.350 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-03-26T06:25:42.351 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-03-26T06:25:42.351 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-03-26T06:25:42.351 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-03-26T06:25:42.351 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-03-26T06:25:42.351 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /root 2024-03-26T06:25:42.351 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-03-26T06:25:42.351 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:25:42.522 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 133/198 2024-03-26T06:25:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 133/198 2024-03-26T06:25:42.537 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:42.537 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T06:25:42.537 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:25:42.816 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-26T06:25:42.840 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 149/198 2024-03-26T06:25:42.862 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 149/198 2024-03-26T06:25:42.862 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:42.862 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T06:25:42.862 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T06:25:42.862 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T06:25:42.862 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:42.907 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1964.g7e98f 150/198 2024-03-26T06:25:42.929 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1964.g7e98f 150/198 2024-03-26T06:25:42.929 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:42.929 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T06:25:42.929 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:42.974 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 151/198 2024-03-26T06:25:42.996 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 151/198 2024-03-26T06:25:42.996 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:42.996 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-26T06:25:42.996 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-26T06:25:42.996 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-26T06:25:42.996 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:43.206 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 130/198 2024-03-26T06:25:43.249 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 130/198 2024-03-26T06:25:43.316 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 152/198 2024-03-26T06:25:43.338 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 152/198 2024-03-26T06:25:43.338 INFO:teuthology.orchestra.run.smithi038.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:43.338 INFO:teuthology.orchestra.run.smithi038.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T06:25:43.338 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T06:25:43.339 INFO:teuthology.orchestra.run.smithi038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T06:25:43.339 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:43.784 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 130/198 2024-03-26T06:25:43.819 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 131/198 2024-03-26T06:25:43.900 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 131/198 2024-03-26T06:25:43.900 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T06:25:43.900 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T06:25:43.900 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:25:43.928 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 132/198 2024-03-26T06:25:44.010 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 134/198 2024-03-26T06:25:44.035 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 134/198 2024-03-26T06:25:44.035 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:44.035 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T06:25:44.035 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T06:25:44.035 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T06:25:44.036 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:25:44.253 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 135/198 2024-03-26T06:25:44.275 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 135/198 2024-03-26T06:25:44.275 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:44.275 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T06:25:44.275 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T06:25:44.275 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T06:25:44.276 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:25:44.712 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 136/198 2024-03-26T06:25:44.734 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 136/198 2024-03-26T06:25:44.734 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:44.734 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T06:25:44.735 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T06:25:44.735 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T06:25:44.735 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:25:44.838 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-26T06:25:45.280 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 138/198 2024-03-26T06:25:45.312 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 138/198 2024-03-26T06:25:45.344 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1964.g7e98f5f6.e 139/198 2024-03-26T06:25:45.403 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1964.g7e98f5f6. 140/198 2024-03-26T06:25:45.426 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noa 141/198 2024-03-26T06:25:45.660 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noa 141/198 2024-03-26T06:25:45.686 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.n 142/198 2024-03-26T06:25:46.648 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.n 142/198 2024-03-26T06:25:46.669 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e9 143/198 2024-03-26T06:25:46.769 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e9 143/198 2024-03-26T06:25:46.934 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1964.g7e98f5f6.el 144/198 2024-03-26T06:25:46.971 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 145/198 2024-03-26T06:25:46.994 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 145/198 2024-03-26T06:25:46.995 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:46.995 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T06:25:46.995 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T06:25:46.995 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T06:25:46.995 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:25:47.039 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 146/198 2024-03-26T06:25:47.050 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 146/198 2024-03-26T06:25:47.095 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:19.0.0-1964.g7e98f5f6.el9.x86_64 147/198 2024-03-26T06:25:47.956 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-26T06:25:47.991 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 149/198 2024-03-26T06:25:48.013 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 149/198 2024-03-26T06:25:48.013 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:48.013 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T06:25:48.013 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T06:25:48.014 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T06:25:48.014 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:25:48.047 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:19.0.0-1964.g7e98f5f6.el9.x86_64 153/198 2024-03-26T06:25:48.058 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1964.g7e98f 150/198 2024-03-26T06:25:48.080 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1964.g7e98f 150/198 2024-03-26T06:25:48.080 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:48.080 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T06:25:48.080 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:25:48.088 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 154/198 2024-03-26T06:25:48.118 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 155/198 2024-03-26T06:25:48.125 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 151/198 2024-03-26T06:25:48.145 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-26T06:25:48.147 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 151/198 2024-03-26T06:25:48.147 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:48.147 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-26T06:25:48.147 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-26T06:25:48.148 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-26T06:25:48.148 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:25:48.336 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-shell-2:19.0.0-1964.g7e98f5f6.el9.x86_64 157/198 2024-03-26T06:25:48.451 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 152/198 2024-03-26T06:25:48.465 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-26T06:25:48.473 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 152/198 2024-03-26T06:25:48.473 INFO:teuthology.orchestra.run.smithi121.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:48.473 INFO:teuthology.orchestra.run.smithi121.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T06:25:48.474 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T06:25:48.474 INFO:teuthology.orchestra.run.smithi121.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T06:25:48.474 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:25:48.511 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 159/198 2024-03-26T06:25:48.569 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-26T06:25:48.618 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-26T06:25:48.661 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-26T06:25:48.694 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_ 163/198 2024-03-26T06:25:48.734 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-26T06:25:48.764 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-26T06:25:48.827 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-26T06:25:48.885 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-26T06:25:48.931 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-26T06:25:49.569 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-26T06:25:49.604 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-26T06:25:49.624 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-26T06:25:50.500 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-top-2:19.0.0-1964.g7e98f5f6.el9.noarch 170/198 2024-03-26T06:25:50.583 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-26T06:25:50.625 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-26T06:25:50.710 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-26T06:25:50.744 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-26T06:25:50.769 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-26T06:25:50.795 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-26T06:25:50.821 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-26T06:25:50.821 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-26T06:25:50.909 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-26T06:25:50.939 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-26T06:25:50.939 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-26T06:25:51.015 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-26T06:25:51.041 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-26T06:25:51.075 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-26T06:25:51.100 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-26T06:25:51.125 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-26T06:25:51.161 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-26T06:25:51.207 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-26T06:25:51.238 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-26T06:25:51.263 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-26T06:25:51.288 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-26T06:25:51.314 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-26T06:25:51.339 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-26T06:25:51.346 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-26T06:25:51.346 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-26T06:25:51.533 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-26T06:25:51.534 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-26T06:25:51.608 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-26T06:25:51.633 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-26T06:25:51.664 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-26T06:25:51.692 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-26T06:25:51.726 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-26T06:25:51.795 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-26T06:25:52.916 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:19.0.0-1964.g7e98f5f6.el9.x86_64 153/198 2024-03-26T06:25:52.957 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 154/198 2024-03-26T06:25:52.987 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 155/198 2024-03-26T06:25:53.014 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-26T06:25:53.209 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-shell-2:19.0.0-1964.g7e98f5f6.el9.x86_64 157/198 2024-03-26T06:25:53.323 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-26T06:25:53.363 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 159/198 2024-03-26T06:25:53.402 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-26T06:25:53.454 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-26T06:25:53.489 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-26T06:25:53.522 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_ 163/198 2024-03-26T06:25:53.562 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-26T06:25:53.610 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-26T06:25:53.664 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-26T06:25:53.725 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-26T06:25:53.758 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-26T06:25:54.336 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-26T06:25:54.372 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-26T06:25:54.392 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-26T06:25:55.211 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-top-2:19.0.0-1964.g7e98f5f6.el9.noarch 170/198 2024-03-26T06:25:55.284 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-26T06:25:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-26T06:25:55.406 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-26T06:25:55.439 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-26T06:25:55.463 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-26T06:25:55.489 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-26T06:25:55.515 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-26T06:25:55.515 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-26T06:25:55.603 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-26T06:25:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-26T06:25:55.633 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-26T06:25:55.717 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-26T06:25:55.735 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 132/198 2024-03-26T06:25:55.736 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /sys 2024-03-26T06:25:55.736 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /proc 2024-03-26T06:25:55.736 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /mnt 2024-03-26T06:25:55.736 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /var/tmp 2024-03-26T06:25:55.736 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /home 2024-03-26T06:25:55.736 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /root 2024-03-26T06:25:55.736 INFO:teuthology.orchestra.run.smithi170.stdout:skipping the directory /tmp 2024-03-26T06:25:55.736 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:25:55.743 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-26T06:25:55.777 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-26T06:25:55.802 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-26T06:25:55.819 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-26T06:25:55.852 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-26T06:25:55.891 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-26T06:25:55.900 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 133/198 2024-03-26T06:25:55.915 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 133/198 2024-03-26T06:25:55.915 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:55.915 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T06:25:55.916 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:25:55.924 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-26T06:25:55.949 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-26T06:25:55.975 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-26T06:25:56.000 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-26T06:25:56.034 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-26T06:25:56.034 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-26T06:25:56.034 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-26T06:25:56.211 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-26T06:25:56.211 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-26T06:25:56.294 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-26T06:25:56.319 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-26T06:25:56.351 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-26T06:25:56.378 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-26T06:25:56.411 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-26T06:25:56.475 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-26T06:25:56.586 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:19.0.0-1964.g7e98f5f6.el9.x86_64 1/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 3/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 4/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1964.g7e98f 5/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 6/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 7/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 8/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 9/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 10/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 11/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:19.0.0-1964.g7e98f5f6.el9.x86_64 12/198 2024-03-26T06:25:56.587 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 13/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-shell-2:19.0.0-1964.g7e98f5f6.el9.x86_64 14/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_ 15/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 16/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 17/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_6 18/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86 19/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 20/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1964.g7e98f5f6.el 21/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:19.0.0-1964.g7e98f5f6.el9. 22/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:19.0.0-1964.g7e98f5f6.el9.x86_6 23/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:19.0.0-1964.g7e98f5f6.el9.x86_64 24/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 25/198 2024-03-26T06:25:56.588 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 26/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 27/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 28/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 29/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1964.g7e98f5f6. 30/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noa 31/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.n 32/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e9 33/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1964.g7e98f5f6.el 34/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 35/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1964.g7e98f5f6.e 36/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 37/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 38/198 2024-03-26T06:25:56.589 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-top-2:19.0.0-1964.g7e98f5f6.el9.noarch 39/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-26T06:25:56.590 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-26T06:25:56.591 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-26T06:25:56.592 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-26T06:25:56.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-26T06:25:56.594 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-26T06:25:56.595 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-26T06:25:56.596 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-26T06:25:56.597 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 151/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 153/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-26T06:25:56.598 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-26T06:25:56.599 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-26T06:25:56.600 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-26T06:25:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-26T06:25:56.601 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-26T06:25:57.403 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 134/198 2024-03-26T06:25:57.429 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 134/198 2024-03-26T06:25:57.429 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:57.429 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T06:25:57.429 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T06:25:57.429 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T06:25:57.429 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:25:57.638 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 135/198 2024-03-26T06:25:57.660 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 135/198 2024-03-26T06:25:57.660 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:57.660 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T06:25:57.660 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T06:25:57.660 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T06:25:57.660 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:25:58.113 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 136/198 2024-03-26T06:25:58.136 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 136/198 2024-03-26T06:25:58.136 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:25:58.136 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T06:25:58.136 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T06:25:58.136 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T06:25:58.136 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:25:58.232 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-26T06:25:58.683 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 138/198 2024-03-26T06:25:58.713 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 138/198 2024-03-26T06:25:58.745 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1964.g7e98f5f6.e 139/198 2024-03-26T06:25:58.806 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1964.g7e98f5f6. 140/198 2024-03-26T06:25:58.828 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noa 141/198 2024-03-26T06:25:59.064 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noa 141/198 2024-03-26T06:25:59.096 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.n 142/198 2024-03-26T06:25:59.344 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-26T06:25:59.344 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:59.344 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2024-03-26T06:25:59.344 INFO:teuthology.orchestra.run.smithi038.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-26T06:25:59.344 INFO:teuthology.orchestra.run.smithi038.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-26T06:25:59.344 INFO:teuthology.orchestra.run.smithi038.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-26T06:25:59.344 INFO:teuthology.orchestra.run.smithi038.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-26T06:25:59.344 INFO:teuthology.orchestra.run.smithi038.stdout: acl-2.3.1-4.el9.x86_64 2024-03-26T06:25:59.344 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-26T06:25:59.344 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-26T06:25:59.344 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-26T06:25:59.345 INFO:teuthology.orchestra.run.smithi038.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: automake-1.16.2-8.el9.noarch 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: bison-3.7.4-5.el9.x86_64 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:25:59.346 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.347 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.347 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.347 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:25:59.347 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:25:59.347 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:25:59.347 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:25:59.347 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:25:59.347 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.348 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.348 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:25:59.348 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.348 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.348 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.348 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:25:59.348 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:25:59.349 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.349 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-shell-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.349 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:25:59.349 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-26T06:25:59.349 INFO:teuthology.orchestra.run.smithi038.stdout: flex-2.6.4-9.el9.x86_64 2024-03-26T06:25:59.349 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T06:25:59.349 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T06:25:59.349 INFO:teuthology.orchestra.run.smithi038.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T06:25:59.349 INFO:teuthology.orchestra.run.smithi038.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: jq-1.6-16.el9.x86_64 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T06:25:59.350 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T06:25:59.351 INFO:teuthology.orchestra.run.smithi038.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T06:25:59.352 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T06:25:59.352 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T06:25:59.352 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T06:25:59.352 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-26T06:25:59.352 INFO:teuthology.orchestra.run.smithi038.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T06:25:59.352 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-13.14-1.el9.x86_64 2024-03-26T06:25:59.352 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-26T06:25:59.352 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T06:25:59.353 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T06:25:59.354 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T06:25:59.354 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-26T06:25:59.354 INFO:teuthology.orchestra.run.smithi038.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-26T06:25:59.354 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T06:25:59.354 INFO:teuthology.orchestra.run.smithi038.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T06:25:59.354 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T06:25:59.354 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T06:25:59.354 INFO:teuthology.orchestra.run.smithi038.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T06:25:59.354 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T06:25:59.354 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T06:25:59.355 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-26T06:25:59.356 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T06:25:59.357 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.357 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.357 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T06:25:59.357 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T06:25:59.357 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T06:25:59.357 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.357 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T06:25:59.357 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T06:25:59.357 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T06:25:59.358 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T06:25:59.358 INFO:teuthology.orchestra.run.smithi038.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T06:25:59.358 INFO:teuthology.orchestra.run.smithi038.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T06:25:59.358 INFO:teuthology.orchestra.run.smithi038.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T06:25:59.358 INFO:teuthology.orchestra.run.smithi038.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-26T06:25:59.358 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T06:25:59.358 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-26T06:25:59.359 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-26T06:25:59.360 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T06:25:59.360 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-26T06:25:59.360 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:25:59.360 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2024-03-26T06:25:59.848 DEBUG:teuthology.parallel:result is None 2024-03-26T06:26:00.051 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.n 142/198 2024-03-26T06:26:00.080 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e9 143/198 2024-03-26T06:26:00.179 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e9 143/198 2024-03-26T06:26:00.350 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1964.g7e98f5f6.el 144/198 2024-03-26T06:26:00.382 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 145/198 2024-03-26T06:26:00.404 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 145/198 2024-03-26T06:26:00.405 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:26:00.405 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T06:26:00.405 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T06:26:00.405 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T06:26:00.405 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:26:00.449 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 146/198 2024-03-26T06:26:00.460 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 146/198 2024-03-26T06:26:00.505 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-2:19.0.0-1964.g7e98f5f6.el9.x86_64 147/198 2024-03-26T06:26:01.285 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-26T06:26:01.285 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:19.0.0-1964.g7e98f5f6.el9.x86_64 1/198 2024-03-26T06:26:01.285 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2/198 2024-03-26T06:26:01.285 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 3/198 2024-03-26T06:26:01.285 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 4/198 2024-03-26T06:26:01.285 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1964.g7e98f 5/198 2024-03-26T06:26:01.285 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 6/198 2024-03-26T06:26:01.285 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 7/198 2024-03-26T06:26:01.285 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 8/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 9/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 10/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 11/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:19.0.0-1964.g7e98f5f6.el9.x86_64 12/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 13/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-shell-2:19.0.0-1964.g7e98f5f6.el9.x86_64 14/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_ 15/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 16/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 17/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_6 18/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86 19/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 20/198 2024-03-26T06:26:01.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1964.g7e98f5f6.el 21/198 2024-03-26T06:26:01.287 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:19.0.0-1964.g7e98f5f6.el9. 22/198 2024-03-26T06:26:01.287 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:19.0.0-1964.g7e98f5f6.el9.x86_6 23/198 2024-03-26T06:26:01.287 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:19.0.0-1964.g7e98f5f6.el9.x86_64 24/198 2024-03-26T06:26:01.287 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 25/198 2024-03-26T06:26:01.287 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 26/198 2024-03-26T06:26:01.287 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 27/198 2024-03-26T06:26:01.287 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 28/198 2024-03-26T06:26:01.302 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 29/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1964.g7e98f5f6. 30/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noa 31/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.n 32/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e9 33/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1964.g7e98f5f6.el 34/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 35/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1964.g7e98f5f6.e 36/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 37/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 38/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-top-2:19.0.0-1964.g7e98f5f6.el9.noarch 39/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-26T06:26:01.303 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-26T06:26:01.304 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-26T06:26:01.305 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-26T06:26:01.306 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-26T06:26:01.307 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-26T06:26:01.308 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-26T06:26:01.309 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-26T06:26:01.310 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-26T06:26:01.311 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 151/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 153/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-26T06:26:01.312 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-26T06:26:01.313 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-26T06:26:01.314 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-26T06:26:01.315 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-26T06:26:01.315 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-26T06:26:01.315 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-26T06:26:01.315 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-26T06:26:01.315 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-26T06:26:01.315 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-26T06:26:01.315 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-26T06:26:01.315 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-26T06:26:01.315 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-26T06:26:01.315 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-26T06:26:01.365 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-26T06:26:01.393 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 149/198 2024-03-26T06:26:01.415 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 149/198 2024-03-26T06:26:01.415 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:26:01.416 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T06:26:01.416 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T06:26:01.416 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T06:26:01.416 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:26:01.460 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1964.g7e98f 150/198 2024-03-26T06:26:01.482 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1964.g7e98f 150/198 2024-03-26T06:26:01.482 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:26:01.482 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T06:26:01.482 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:26:01.528 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 151/198 2024-03-26T06:26:01.550 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 151/198 2024-03-26T06:26:01.550 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:26:01.550 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-26T06:26:01.550 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-26T06:26:01.551 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-26T06:26:01.551 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:26:01.861 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 152/198 2024-03-26T06:26:01.883 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 152/198 2024-03-26T06:26:01.884 INFO:teuthology.orchestra.run.smithi170.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T06:26:01.884 INFO:teuthology.orchestra.run.smithi170.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T06:26:01.884 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T06:26:01.884 INFO:teuthology.orchestra.run.smithi170.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T06:26:01.884 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:26:04.372 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-26T06:26:04.373 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:26:04.373 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-03-26T06:26:04.373 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-26T06:26:04.373 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-26T06:26:04.373 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-26T06:26:04.373 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-26T06:26:04.373 INFO:teuthology.orchestra.run.smithi121.stdout: acl-2.3.1-4.el9.x86_64 2024-03-26T06:26:04.374 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-26T06:26:04.374 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-26T06:26:04.374 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-26T06:26:04.374 INFO:teuthology.orchestra.run.smithi121.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-26T06:26:04.374 INFO:teuthology.orchestra.run.smithi121.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-26T06:26:04.374 INFO:teuthology.orchestra.run.smithi121.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-26T06:26:04.374 INFO:teuthology.orchestra.run.smithi121.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-26T06:26:04.374 INFO:teuthology.orchestra.run.smithi121.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-26T06:26:04.374 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.375 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.375 INFO:teuthology.orchestra.run.smithi121.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-26T06:26:04.375 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-26T06:26:04.375 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-26T06:26:04.375 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-26T06:26:04.375 INFO:teuthology.orchestra.run.smithi121.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-26T06:26:04.375 INFO:teuthology.orchestra.run.smithi121.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-26T06:26:04.375 INFO:teuthology.orchestra.run.smithi121.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-26T06:26:04.375 INFO:teuthology.orchestra.run.smithi121.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-26T06:26:04.375 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-26T06:26:04.375 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-03-26T06:26:04.376 INFO:teuthology.orchestra.run.smithi121.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T06:26:04.376 INFO:teuthology.orchestra.run.smithi121.stdout: automake-1.16.2-8.el9.noarch 2024-03-26T06:26:04.376 INFO:teuthology.orchestra.run.smithi121.stdout: bison-3.7.4-5.el9.x86_64 2024-03-26T06:26:04.376 INFO:teuthology.orchestra.run.smithi121.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T06:26:04.376 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.376 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.376 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.376 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.377 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:04.377 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.377 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.377 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.377 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:04.377 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:04.378 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:04.378 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:04.378 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:04.378 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.378 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.378 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:04.378 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.378 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.378 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.378 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:04.379 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:04.379 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.379 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-shell-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.379 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:04.379 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-26T06:26:04.379 INFO:teuthology.orchestra.run.smithi121.stdout: flex-2.6.4-9.el9.x86_64 2024-03-26T06:26:04.379 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T06:26:04.379 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T06:26:04.379 INFO:teuthology.orchestra.run.smithi121.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T06:26:04.379 INFO:teuthology.orchestra.run.smithi121.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-26T06:26:04.380 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T06:26:04.380 INFO:teuthology.orchestra.run.smithi121.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T06:26:04.380 INFO:teuthology.orchestra.run.smithi121.stdout: jq-1.6-16.el9.x86_64 2024-03-26T06:26:04.380 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-26T06:26:04.380 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T06:26:04.381 INFO:teuthology.orchestra.run.smithi121.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T06:26:04.381 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.381 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.381 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.381 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T06:26:04.381 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T06:26:04.382 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T06:26:04.382 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T06:26:04.382 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T06:26:04.382 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.382 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.382 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T06:26:04.382 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.382 INFO:teuthology.orchestra.run.smithi121.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-26T06:26:04.382 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T06:26:04.382 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-26T06:26:04.382 INFO:teuthology.orchestra.run.smithi121.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-13.14-1.el9.x86_64 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-26T06:26:04.383 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-26T06:26:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-26T06:26:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T06:26:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T06:26:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T06:26:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-26T06:26:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T06:26:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T06:26:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T06:26:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.384 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T06:26:04.385 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T06:26:04.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T06:26:04.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T06:26:04.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T06:26:04.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T06:26:04.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T06:26:04.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T06:26:04.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T06:26:04.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T06:26:04.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T06:26:04.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T06:26:04.386 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T06:26:04.387 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T06:26:04.388 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T06:26:04.388 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T06:26:04.388 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T06:26:04.388 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-26T06:26:04.388 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T06:26:04.388 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.388 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.388 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T06:26:04.388 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T06:26:04.388 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T06:26:04.388 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T06:26:04.389 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-26T06:26:04.390 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T06:26:04.391 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-26T06:26:04.391 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-03-26T06:26:04.391 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-03-26T06:26:04.862 DEBUG:teuthology.parallel:result is None 2024-03-26T06:26:06.292 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-test-2:19.0.0-1964.g7e98f5f6.el9.x86_64 153/198 2024-03-26T06:26:06.333 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 154/198 2024-03-26T06:26:06.363 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : rbd-nbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 155/198 2024-03-26T06:26:06.398 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-26T06:26:06.586 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-shell-2:19.0.0-1964.g7e98f5f6.el9.x86_64 157/198 2024-03-26T06:26:06.700 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 158/198 2024-03-26T06:26:06.748 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : ceph-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 159/198 2024-03-26T06:26:06.788 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-26T06:26:06.832 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-26T06:26:06.876 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-26T06:26:06.908 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libcephfs-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_ 163/198 2024-03-26T06:26:06.949 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-26T06:26:06.978 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-26T06:26:07.058 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-26T06:26:07.120 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-26T06:26:07.152 INFO:teuthology.orchestra.run.smithi170.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-26T06:26:07.730 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-26T06:26:07.775 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-26T06:26:07.795 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 169/198 2024-03-26T06:26:08.606 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : cephfs-top-2:19.0.0-1964.g7e98f5f6.el9.noarch 170/198 2024-03-26T06:26:08.688 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-26T06:26:08.728 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-26T06:26:08.808 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-26T06:26:08.875 INFO:teuthology.orchestra.run.smithi170.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-26T06:26:08.908 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-26T06:26:08.942 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-26T06:26:08.968 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-26T06:26:08.968 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-26T06:26:09.057 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-26T06:26:09.086 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-26T06:26:09.086 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-26T06:26:09.171 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-26T06:26:09.197 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-26T06:26:09.230 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-26T06:26:09.264 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-26T06:26:09.289 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-26T06:26:09.322 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-26T06:26:09.362 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-26T06:26:09.394 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-26T06:26:09.419 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-26T06:26:09.453 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-26T06:26:09.478 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-26T06:26:09.504 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-26T06:26:09.510 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-26T06:26:09.510 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-26T06:26:09.690 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-26T06:26:09.690 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-26T06:26:09.765 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-26T06:26:09.789 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-26T06:26:09.821 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-26T06:26:09.857 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-26T06:26:09.882 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-26T06:26:09.946 INFO:teuthology.orchestra.run.smithi170.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-26T06:26:14.860 INFO:teuthology.orchestra.run.smithi170.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-26T06:26:14.860 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-2:19.0.0-1964.g7e98f5f6.el9.x86_64 1/198 2024-03-26T06:26:14.860 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2/198 2024-03-26T06:26:14.860 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 3/198 2024-03-26T06:26:14.860 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 4/198 2024-03-26T06:26:14.860 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1964.g7e98f 5/198 2024-03-26T06:26:14.860 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 6/198 2024-03-26T06:26:14.860 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 7/198 2024-03-26T06:26:14.860 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 8/198 2024-03-26T06:26:14.860 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 9/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 10/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 11/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-test-2:19.0.0-1964.g7e98f5f6.el9.x86_64 12/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 13/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-shell-2:19.0.0-1964.g7e98f5f6.el9.x86_64 14/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_ 15/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 16/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 17/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_6 18/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86 19/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 20/198 2024-03-26T06:26:14.861 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1964.g7e98f5f6.el 21/198 2024-03-26T06:26:14.862 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ceph-common-2:19.0.0-1964.g7e98f5f6.el9. 22/198 2024-03-26T06:26:14.862 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cephfs-2:19.0.0-1964.g7e98f5f6.el9.x86_6 23/198 2024-03-26T06:26:14.862 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rados-2:19.0.0-1964.g7e98f5f6.el9.x86_64 24/198 2024-03-26T06:26:14.862 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 25/198 2024-03-26T06:26:14.862 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 26/198 2024-03-26T06:26:14.862 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 27/198 2024-03-26T06:26:14.878 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 28/198 2024-03-26T06:26:14.878 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : rbd-nbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 29/198 2024-03-26T06:26:14.878 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1964.g7e98f5f6. 30/198 2024-03-26T06:26:14.878 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noa 31/198 2024-03-26T06:26:14.878 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.n 32/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e9 33/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1964.g7e98f5f6.el 34/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 35/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1964.g7e98f5f6.e 36/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 37/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 38/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : cephfs-top-2:19.0.0-1964.g7e98f5f6.el9.noarch 39/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-26T06:26:14.879 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-26T06:26:14.880 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 58/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 59/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/198 2024-03-26T06:26:14.881 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 70/198 2024-03-26T06:26:14.882 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 71/198 2024-03-26T06:26:14.882 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 72/198 2024-03-26T06:26:14.882 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 73/198 2024-03-26T06:26:14.882 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 74/198 2024-03-26T06:26:14.882 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 75/198 2024-03-26T06:26:14.882 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 76/198 2024-03-26T06:26:14.882 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 77/198 2024-03-26T06:26:14.882 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 78/198 2024-03-26T06:26:14.882 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 79/198 2024-03-26T06:26:14.882 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 80/198 2024-03-26T06:26:14.882 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 81/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 82/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 83/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 84/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-ply-3.11-14.el9.noarch 85/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 86/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 87/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : jq-1.6-16.el9.x86_64 88/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 89/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 90/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 91/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 92/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 93/198 2024-03-26T06:26:14.883 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 94/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-13.14-1.el9.x86_64 95/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 96/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 97/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 98/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 101/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 102/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 103/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-26T06:26:14.884 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-26T06:26:14.885 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-26T06:26:14.886 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-26T06:26:14.887 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 151/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 153/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-26T06:26:14.888 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-26T06:26:14.889 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-26T06:26:14.890 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-26T06:26:14.891 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-26T06:26:14.892 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-26T06:26:14.892 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-26T06:26:18.042 INFO:teuthology.orchestra.run.smithi170.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout:Upgraded: 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout: acl-2.3.1-4.el9.x86_64 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-26T06:26:18.043 INFO:teuthology.orchestra.run.smithi170.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: librados2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: librbd1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-26T06:26:18.044 INFO:teuthology.orchestra.run.smithi170.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout:Installed: 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: automake-1.16.2-8.el9.noarch 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: bison-3.7.4-5.el9.x86_64 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-base-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-grafana-dashboards-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-immutable-object-cache-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.045 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mds-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-dashboard-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-modules-core-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mgr-rook-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-mon-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-osd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-prometheus-alerts-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-radosgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-selinux-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.046 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-test-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: ceph-volume-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: cephadm-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-shell-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: cephfs-top-2:19.0.0-1964.g7e98f5f6.el9.noarch 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: flex-2.6.4-9.el9.x86_64 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T06:26:18.047 INFO:teuthology.orchestra.run.smithi170.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: jq-1.6-16.el9.x86_64 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: libcephfs2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: libcephsqlite-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T06:26:18.048 INFO:teuthology.orchestra.run.smithi170.stdout: librados-devel-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: libradosstriper1-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: librgw2-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T06:26:18.049 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-13.14-1.el9.x86_64 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: postgresql-server-13.14-1.el9.x86_64 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-26T06:26:18.050 INFO:teuthology.orchestra.run.smithi170.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-argparse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ceph-common-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cephfs-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T06:26:18.051 INFO:teuthology.orchestra.run.smithi170.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T06:26:18.052 INFO:teuthology.orchestra.run.smithi170.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T06:26:18.052 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T06:26:18.052 INFO:teuthology.orchestra.run.smithi170.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T06:26:18.052 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T06:26:18.052 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T06:26:18.052 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T06:26:18.052 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T06:26:18.052 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T06:26:18.053 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rados-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.054 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rgw-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T06:26:18.055 INFO:teuthology.orchestra.run.smithi170.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-fuse-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-mirror-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: rbd-nbd-2:19.0.0-1964.g7e98f5f6.el9.x86_64 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T06:26:18.056 INFO:teuthology.orchestra.run.smithi170.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T06:26:18.057 INFO:teuthology.orchestra.run.smithi170.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-26T06:26:18.057 INFO:teuthology.orchestra.run.smithi170.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-26T06:26:18.057 INFO:teuthology.orchestra.run.smithi170.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T06:26:18.057 INFO:teuthology.orchestra.run.smithi170.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-26T06:26:18.057 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:26:18.057 INFO:teuthology.orchestra.run.smithi170.stdout:Complete! 2024-03-26T06:26:18.540 DEBUG:teuthology.parallel:result is None 2024-03-26T06:26:18.540 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:26:18.687 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T06:26:18.712 INFO:teuthology.orchestra.run.smithi038.stdout:19.0.0-1964.g7e98f5f6.el9 2024-03-26T06:26:18.713 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1964.g7e98f5f6.el9 2024-03-26T06:26:18.713 INFO:teuthology.task.install:The correct ceph version 19.0.0-1964.g7e98f5f6 is installed. 2024-03-26T06:26:18.715 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:26:18.968 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T06:26:18.993 INFO:teuthology.orchestra.run.smithi121.stdout:19.0.0-1964.g7e98f5f6.el9 2024-03-26T06:26:18.994 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1964.g7e98f5f6.el9 2024-03-26T06:26:18.994 INFO:teuthology.task.install:The correct ceph version 19.0.0-1964.g7e98f5f6 is installed. 2024-03-26T06:26:18.996 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:26:19.275 DEBUG:teuthology.orchestra.run.smithi170:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T06:26:19.300 INFO:teuthology.orchestra.run.smithi170.stdout:19.0.0-1964.g7e98f5f6.el9 2024-03-26T06:26:19.301 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1964.g7e98f5f6.el9 2024-03-26T06:26:19.301 INFO:teuthology.task.install:The correct ceph version 19.0.0-1964.g7e98f5f6 is installed. 2024-03-26T06:26:19.303 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T06:26:19.304 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:26:19.304 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T06:26:19.339 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-26T06:26:19.339 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T06:26:19.373 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-26T06:26:19.374 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T06:26:19.403 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T06:26:19.404 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:26:19.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T06:26:19.435 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T06:26:19.507 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-26T06:26:19.508 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T06:26:19.540 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T06:26:19.610 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-26T06:26:19.610 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T06:26:19.638 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T06:26:19.705 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T06:26:19.705 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:26:19.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T06:26:19.737 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T06:26:19.807 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-26T06:26:19.807 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T06:26:19.835 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T06:26:19.902 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-26T06:26:19.902 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T06:26:19.929 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T06:26:19.995 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T06:26:19.996 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:26:19.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T06:26:20.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T06:26:20.090 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-26T06:26:20.090 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T06:26:20.117 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T06:26:20.184 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-26T06:26:20.184 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T06:26:20.211 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T06:26:20.278 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T06:26:20.403 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'SLOW_OPS', 'slow request'], 'sha1': '7e98f5f6b535defc7fa715b456904ffc0500aa9f', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-26T06:26:20.404 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:26:20.404 INFO:tasks.cephadm:Cluster fsid is c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:26:20.404 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T06:26:20.404 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.38', 'mon.b': '172.21.15.121', 'mon.c': '172.21.15.170'} 2024-03-26T06:26:20.405 INFO:tasks.cephadm:First mon is mon.a on smithi038 2024-03-26T06:26:20.405 INFO:tasks.cephadm:First mgr is x 2024-03-26T06:26:20.405 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T06:26:20.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2024-03-26T06:26:20.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname $(hostname -s) 2024-03-26T06:26:20.463 DEBUG:teuthology.orchestra.run.smithi170:> sudo hostname $(hostname -s) 2024-03-26T06:26:20.490 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-26T06:26:20.490 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:26:20.635 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '7e98f5f6b535defc7fa715b456904ffc0500aa9f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77904/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1964-g7e98f5f6', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1964.g7e98f5f6'}, 'url': 'https://4.chacra.ceph.com/r/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-23 11:54:09.441197', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'rishabh-2024mar8-2', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-26T06:26:20.857 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref rishabh-2024mar8-2, sha1 7e98f5f6b535defc7fa715b456904ffc0500aa9f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:26:20.858 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/x86_64/flavors/default/cephadm 2024-03-26T06:26:20.858 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T06:26:21.493 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-r--r--. 1 ubuntu ubuntu 767477 Mar 26 06:26 /home/ubuntu/cephtest/cephadm 2024-03-26T06:26:21.493 DEBUG:teuthology.orchestra.run.smithi121:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T06:26:21.971 INFO:teuthology.orchestra.run.smithi121.stdout:-rw-r--r--. 1 ubuntu ubuntu 767477 Mar 26 06:26 /home/ubuntu/cephtest/cephadm 2024-03-26T06:26:21.971 DEBUG:teuthology.orchestra.run.smithi170:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/rishabh-2024mar8-2/7e98f5f6b535defc7fa715b456904ffc0500aa9f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T06:26:22.593 INFO:teuthology.orchestra.run.smithi170.stdout:-rw-r--r--. 1 ubuntu ubuntu 767477 Mar 26 06:26 /home/ubuntu/cephtest/cephadm 2024-03-26T06:26:22.594 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T06:26:22.612 DEBUG:teuthology.orchestra.run.smithi121:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T06:26:22.631 DEBUG:teuthology.orchestra.run.smithi170:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T06:26:22.669 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f on all hosts... 2024-03-26T06:26:22.669 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f pull 2024-03-26T06:26:22.671 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f pull 2024-03-26T06:26:22.674 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f pull 2024-03-26T06:26:23.489 INFO:teuthology.orchestra.run.smithi121.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f... 2024-03-26T06:26:23.521 INFO:teuthology.orchestra.run.smithi170.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f... 2024-03-26T06:26:23.541 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f... 2024-03-26T06:26:55.614 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:26:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 19.0.0-1964-g7e98f5f6 (7e98f5f6b535defc7fa715b456904ffc0500aa9f) squid (dev)", 2024-03-26T06:26:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c", 2024-03-26T06:26:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2024-03-26T06:26:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265" 2024-03-26T06:26:55.615 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:26:55.615 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:27:12.964 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-26T06:27:12.964 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_version": "ceph version 19.0.0-1964-g7e98f5f6 (7e98f5f6b535defc7fa715b456904ffc0500aa9f) squid (dev)", 2024-03-26T06:27:12.964 INFO:teuthology.orchestra.run.smithi170.stdout: "image_id": "10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c", 2024-03-26T06:27:12.964 INFO:teuthology.orchestra.run.smithi170.stdout: "repo_digests": [ 2024-03-26T06:27:12.965 INFO:teuthology.orchestra.run.smithi170.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265" 2024-03-26T06:27:12.965 INFO:teuthology.orchestra.run.smithi170.stdout: ] 2024-03-26T06:27:12.965 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-26T06:27:13.603 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-03-26T06:27:13.603 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_version": "ceph version 19.0.0-1964-g7e98f5f6 (7e98f5f6b535defc7fa715b456904ffc0500aa9f) squid (dev)", 2024-03-26T06:27:13.603 INFO:teuthology.orchestra.run.smithi121.stdout: "image_id": "10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c", 2024-03-26T06:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout: "repo_digests": [ 2024-03-26T06:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265" 2024-03-26T06:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout: ] 2024-03-26T06:27:13.604 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-03-26T06:27:13.628 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2024-03-26T06:27:13.663 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph 2024-03-26T06:27:13.695 DEBUG:teuthology.orchestra.run.smithi170:> sudo mkdir -p /etc/ceph 2024-03-26T06:27:13.731 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2024-03-26T06:27:13.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /etc/ceph 2024-03-26T06:27:13.787 DEBUG:teuthology.orchestra.run.smithi170:> sudo chmod 777 /etc/ceph 2024-03-26T06:27:13.817 INFO:tasks.cephadm:Writing seed config... 2024-03-26T06:27:13.819 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-26T06:27:13.819 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-26T06:27:13.819 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-26T06:27:13.819 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-26T06:27:13.819 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-26T06:27:13.819 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-26T06:27:13.819 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-26T06:27:13.819 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-26T06:27:13.819 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-26T06:27:13.819 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-26T06:27:13.820 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T06:27:13.821 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-26T06:27:13.822 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-03-26T06:27:13.822 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-26T06:27:13.822 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-26T06:27:13.822 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-26T06:27:13.822 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-26T06:27:13.822 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-03-26T06:27:13.822 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-26T06:27:13.822 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-26T06:27:13.822 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T06:27:13.822 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-26T06:27:13.823 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:27:13.823 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T06:27:13.839 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c2824556-eb39-11ee-95d0-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-26T06:27:13.840 DEBUG:teuthology.orchestra.run.smithi038:mon.a> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T06:27:13.882 DEBUG:teuthology.orchestra.run.smithi038:mgr.x> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T06:27:13.924 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T06:27:13.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f -v bootstrap --fsid c2824556-eb39-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.38 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T06:27:14.123 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2024-03-26T06:27:14.123 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f', '-v', 'bootstrap', '--fsid', 'c2824556-eb39-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.38', '--skip-admin-label'] 2024-03-26T06:27:14.149 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T06:27:14.149 INFO:teuthology.orchestra.run.smithi038.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T06:27:14.150 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying podman|docker is present... 2024-03-26T06:27:14.173 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T06:27:14.173 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying lvm2 is present... 2024-03-26T06:27:14.174 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying time synchronization is in place... 2024-03-26T06:27:14.181 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T06:27:14.181 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T06:27:14.187 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T06:27:14.187 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T06:27:14.193 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-03-26T06:27:14.199 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-03-26T06:27:14.199 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-03-26T06:27:14.199 INFO:teuthology.orchestra.run.smithi038.stdout:Repeating the final host check... 2024-03-26T06:27:14.221 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T06:27:14.222 INFO:teuthology.orchestra.run.smithi038.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T06:27:14.222 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl is present 2024-03-26T06:27:14.222 INFO:teuthology.orchestra.run.smithi038.stdout:lvcreate is present 2024-03-26T06:27:14.228 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T06:27:14.228 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T06:27:14.233 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T06:27:14.234 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T06:27:14.240 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2024-03-26T06:27:14.246 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2024-03-26T06:27:14.246 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2024-03-26T06:27:14.246 INFO:teuthology.orchestra.run.smithi038.stdout:Host looks OK 2024-03-26T06:27:14.246 INFO:teuthology.orchestra.run.smithi038.stdout:Cluster fsid: c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:27:14.246 INFO:teuthology.orchestra.run.smithi038.stdout:Acquiring lock 140415223676352 on /run/cephadm/c2824556-eb39-11ee-95d0-87774f69a715.lock 2024-03-26T06:27:14.246 INFO:teuthology.orchestra.run.smithi038.stdout:Lock 140415223676352 acquired on /run/cephadm/c2824556-eb39-11ee-95d0-87774f69a715.lock 2024-03-26T06:27:14.247 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 3300 ... 2024-03-26T06:27:14.247 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 6789 ... 2024-03-26T06:27:14.247 INFO:teuthology.orchestra.run.smithi038.stdout:Base mon IP(s) is [172.21.15.38:3300, 172.21.15.38:6789], mon addrv is [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-03-26T06:27:14.250 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.38 metric 100 2024-03-26T06:27:14.250 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.38 metric 100 2024-03-26T06:27:14.253 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T06:27:14.253 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-26T06:27:14.253 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-26T06:27:14.257 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T06:27:14.257 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T06:27:14.257 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T06:27:14.257 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-26T06:27:14.257 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-03-26T06:27:14.257 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T06:27:14.258 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-03-26T06:27:14.258 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2024-03-26T06:27:14.258 INFO:teuthology.orchestra.run.smithi038.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T06:27:14.258 INFO:teuthology.orchestra.run.smithi038.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T06:27:14.260 INFO:teuthology.orchestra.run.smithi038.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f... 2024-03-26T06:27:14.744 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c 2024-03-26T06:27:14.744 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f... 2024-03-26T06:27:14.745 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T06:27:14.745 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:cefe3f08b7d2324bed4b4b268e386dce7ad863cb02a7a7641e64ed611288b7c8 2024-03-26T06:27:14.745 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T06:27:14.745 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying config sha256:10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c 2024-03-26T06:27:14.745 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T06:27:15.683 INFO:teuthology.orchestra.run.smithi038.stdout:ceph: stdout ceph version 19.0.0-1964-g7e98f5f6 (7e98f5f6b535defc7fa715b456904ffc0500aa9f) squid (dev) 2024-03-26T06:27:15.683 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph version: ceph version 19.0.0-1964-g7e98f5f6 (7e98f5f6b535defc7fa715b456904ffc0500aa9f) squid (dev) 2024-03-26T06:27:15.683 INFO:teuthology.orchestra.run.smithi038.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T06:27:16.629 INFO:teuthology.orchestra.run.smithi038.stdout:stat: stdout 167 167 2024-03-26T06:27:16.629 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial keys... 2024-03-26T06:27:17.637 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQDEagJmMSMLNxAAs4yGypfU7ND5XBRel9yjQw== 2024-03-26T06:27:18.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQDFagJmycbkORAAOgUwMcn20E6sPbERQvcxkA== 2024-03-26T06:27:23.238 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQDGagJmQ0I2NxAAxxB9Wgzdhff4CDB/VJWNFQ== 2024-03-26T06:27:23.238 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial monmap... 2024-03-26T06:27:29.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T06:27:29.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T06:27:29.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:27:29.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T06:27:29.610 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool for a [v2:172.21.15.38:3300,v1:172.21.15.38:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T06:27:29.610 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = quincy 2024-03-26T06:27:29.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: set fsid to c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:27:29.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T06:27:29.610 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:27:29.610 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mon... 2024-03-26T06:27:30.940 INFO:teuthology.orchestra.run.smithi038.stdout:create mon.a on 2024-03-26T06:27:31.160 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T06:27:31.304 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T06:27:31.481 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c2824556-eb39-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-c2824556-eb39-11ee-95d0-87774f69a715.target. 2024-03-26T06:27:31.481 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c2824556-eb39-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-c2824556-eb39-11ee-95d0-87774f69a715.target. 2024-03-26T06:27:31.702 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.a 2024-03-26T06:27:31.702 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T06:27:31.860 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c2824556-eb39-11ee-95d0-87774f69a715.target.wants/ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-c2824556-eb39-11ee-95d0-87774f69a715@.service. 2024-03-26T06:27:32.405 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T06:27:32.406 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-26T06:27:32.412 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T06:27:32.412 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T06:27:32.412 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-26T06:27:32.412 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T06:27:32.413 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon to start... 2024-03-26T06:27:32.413 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon... 2024-03-26T06:27:33.391 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T06:27:33.391 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout id: c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout services: 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.49847s) 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout data: 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T06:27:33.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.793+0000 7f960a9ea700 1 Processor -- start 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.793+0000 7f960a9ea700 1 -- start start 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.793+0000 7f960a9ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960410a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.793+0000 7f960a9ea700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960407bdf0 con 0x7f9604108430 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.793+0000 7f9603fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960410a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.793+0000 7f9603fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960410a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35300/0 (socket says 172.21.15.38:35300) 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.793+0000 7f9603fff700 1 -- 172.21.15.38:0/3004871292 learned_addr learned my addr 172.21.15.38:0/3004871292 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.794+0000 7f9603fff700 1 -- 172.21.15.38:0/3004871292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960410ad60 con 0x7f9604108430 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.794+0000 7f9603fff700 1 --2- 172.21.15.38:0/3004871292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960410a820 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95f40098a0 tx=0x7f95f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bcb5609f5cf513c server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.794+0000 7f9602ffd700 1 -- 172.21.15.38:0/3004871292 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95f4004160 con 0x7f9604108430 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.794+0000 7f9602ffd700 1 -- 172.21.15.38:0/3004871292 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f4004300 con 0x7f9604108430 2024-03-26T06:27:33.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.794+0000 7f9602ffd700 1 -- 172.21.15.38:0/3004871292 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95f40036a0 con 0x7f9604108430 2024-03-26T06:27:33.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.794+0000 7f960a9ea700 1 -- 172.21.15.38:0/3004871292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 msgr2=0x7f960410a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:33.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.794+0000 7f960a9ea700 1 --2- 172.21.15.38:0/3004871292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960410a820 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f95f40098a0 tx=0x7f95f4009bb0 comp rx=0 tx=0).stop 2024-03-26T06:27:33.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.795+0000 7f960a9ea700 1 -- 172.21.15.38:0/3004871292 shutdown_connections 2024-03-26T06:27:33.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.795+0000 7f960a9ea700 1 --2- 172.21.15.38:0/3004871292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960410a820 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:33.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.795+0000 7f960a9ea700 1 -- 172.21.15.38:0/3004871292 >> 172.21.15.38:0/3004871292 conn(0x7f9604101e40 msgr2=0x7f9604104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:33.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.795+0000 7f960a9ea700 1 -- 172.21.15.38:0/3004871292 shutdown_connections 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.795+0000 7f960a9ea700 1 -- 172.21.15.38:0/3004871292 wait complete. 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.795+0000 7f960a9ea700 1 Processor -- start 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.795+0000 7f960a9ea700 1 -- start start 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f960a9ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960419b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f960a9ea700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960410b1d0 con 0x7f9604108430 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f9603fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960419b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f9603fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960419b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35312/0 (socket says 172.21.15.38:35312) 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f9603fff700 1 -- 172.21.15.38:0/2555961984 learned_addr learned my addr 172.21.15.38:0/2555961984 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f9603fff700 1 -- 172.21.15.38:0/2555961984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960419b640 con 0x7f9604108430 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f9603fff700 1 --2- 172.21.15.38:0/2555961984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960419b100 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f95f4000c00 tx=0x7f95f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f96017fa700 1 -- 172.21.15.38:0/2555961984 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95f4003db0 con 0x7f9604108430 2024-03-26T06:27:33.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f96017fa700 1 -- 172.21.15.38:0/2555961984 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f4011630 con 0x7f9604108430 2024-03-26T06:27:33.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f960a9ea700 1 -- 172.21.15.38:0/2555961984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960419b8d0 con 0x7f9604108430 2024-03-26T06:27:33.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f96017fa700 1 -- 172.21.15.38:0/2555961984 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95f4011950 con 0x7f9604108430 2024-03-26T06:27:33.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.796+0000 7f960a9ea700 1 -- 172.21.15.38:0/2555961984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960419e570 con 0x7f9604108430 2024-03-26T06:27:33.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.797+0000 7f96017fa700 1 -- 172.21.15.38:0/2555961984 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f95f401a460 con 0x7f9604108430 2024-03-26T06:27:33.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.797+0000 7f96017fa700 1 -- 172.21.15.38:0/2555961984 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f95f401b780 con 0x7f9604108430 2024-03-26T06:27:33.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.797+0000 7f960a9ea700 1 -- 172.21.15.38:0/2555961984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9604068f20 con 0x7f9604108430 2024-03-26T06:27:33.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.800+0000 7f96017fa700 1 -- 172.21.15.38:0/2555961984 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f95f401ba20 con 0x7f9604108430 2024-03-26T06:27:33.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.872+0000 7f960a9ea700 1 -- 172.21.15.38:0/2555961984 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f960402db30 con 0x7f9604108430 2024-03-26T06:27:33.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.872+0000 7f96017fa700 1 -- 172.21.15.38:0/2555961984 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+316 (secure 0 0 0) 0x7f95f401bc00 con 0x7f9604108430 2024-03-26T06:27:33.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.873+0000 7f960a9ea700 1 -- 172.21.15.38:0/2555961984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 msgr2=0x7f960419b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:33.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.873+0000 7f960a9ea700 1 --2- 172.21.15.38:0/2555961984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960419b100 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f95f4000c00 tx=0x7f95f4004750 comp rx=0 tx=0).stop 2024-03-26T06:27:33.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.873+0000 7f960a9ea700 1 -- 172.21.15.38:0/2555961984 shutdown_connections 2024-03-26T06:27:33.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.873+0000 7f960a9ea700 1 --2- 172.21.15.38:0/2555961984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9604108430 0x7f960419b100 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:33.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.873+0000 7f960a9ea700 1 -- 172.21.15.38:0/2555961984 >> 172.21.15.38:0/2555961984 conn(0x7f9604101e40 msgr2=0x7f9604104170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:33.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.873+0000 7f960a9ea700 1 -- 172.21.15.38:0/2555961984 shutdown_connections 2024-03-26T06:27:33.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:32.873+0000 7f960a9ea700 1 -- 172.21.15.38:0/2555961984 wait complete. 2024-03-26T06:27:33.397 INFO:teuthology.orchestra.run.smithi038.stdout:mon is available 2024-03-26T06:27:33.397 INFO:teuthology.orchestra.run.smithi038.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T06:27:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:33 smithi038 ceph-mon[31897]: from='client.? 172.21.15.38:0/2555961984' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T06:27:34.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:34.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T06:27:34.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:27:34.440 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-26T06:27:34.441 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T06:27:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T06:27:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.801+0000 7f7b5aeb2700 1 Processor -- start 2024-03-26T06:27:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.801+0000 7f7b5aeb2700 1 -- start start 2024-03-26T06:27:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.801+0000 7f7b5aeb2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5407c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.801+0000 7f7b5aeb2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5407c890 con 0x7f7b5407de10 2024-03-26T06:27:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.801+0000 7f7b58c4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5407c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:34.442 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.801+0000 7f7b58c4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5407c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35320/0 (socket says 172.21.15.38:35320) 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.801+0000 7f7b58c4e700 1 -- 172.21.15.38:0/1604671679 learned_addr learned my addr 172.21.15.38:0/1604671679 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.801+0000 7f7b58c4e700 1 -- 172.21.15.38:0/1604671679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5407ca10 con 0x7f7b5407de10 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.802+0000 7f7b58c4e700 1 --2- 172.21.15.38:0/1604671679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5407c2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7b480098a0 tx=0x7f7b48009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b52bf917df394b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.802+0000 7f7b537fe700 1 -- 172.21.15.38:0/1604671679 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7b48004160 con 0x7f7b5407de10 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.802+0000 7f7b537fe700 1 -- 172.21.15.38:0/1604671679 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b48004300 con 0x7f7b5407de10 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.802+0000 7f7b537fe700 1 -- 172.21.15.38:0/1604671679 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7b480036a0 con 0x7f7b5407de10 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.802+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/1604671679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 msgr2=0x7f7b5407c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.802+0000 7f7b5aeb2700 1 --2- 172.21.15.38:0/1604671679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5407c2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f7b480098a0 tx=0x7f7b48009bb0 comp rx=0 tx=0).stop 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.802+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/1604671679 shutdown_connections 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.802+0000 7f7b5aeb2700 1 --2- 172.21.15.38:0/1604671679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5407c2c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.802+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/1604671679 >> 172.21.15.38:0/1604671679 conn(0x7f7b54102fd0 msgr2=0x7f7b541053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.802+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/1604671679 shutdown_connections 2024-03-26T06:27:34.443 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.802+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/1604671679 wait complete. 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.803+0000 7f7b5aeb2700 1 Processor -- start 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.803+0000 7f7b5aeb2700 1 -- start start 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.803+0000 7f7b5aeb2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5419d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.803+0000 7f7b5aeb2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5407cf40 con 0x7f7b5407de10 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.803+0000 7f7b58c4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5419d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.803+0000 7f7b58c4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5419d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35330/0 (socket says 172.21.15.38:35330) 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.803+0000 7f7b58c4e700 1 -- 172.21.15.38:0/2684383177 learned_addr learned my addr 172.21.15.38:0/2684383177 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.804+0000 7f7b58c4e700 1 -- 172.21.15.38:0/2684383177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5419d850 con 0x7f7b5407de10 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.804+0000 7f7b58c4e700 1 --2- 172.21.15.38:0/2684383177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5419d310 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7b48000c00 tx=0x7f7b48004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.804+0000 7f7b51ffb700 1 -- 172.21.15.38:0/2684383177 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7b48003db0 con 0x7f7b5407de10 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.804+0000 7f7b51ffb700 1 -- 172.21.15.38:0/2684383177 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b48011630 con 0x7f7b5407de10 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.804+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/2684383177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b5419dae0 con 0x7f7b5407de10 2024-03-26T06:27:34.444 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.804+0000 7f7b51ffb700 1 -- 172.21.15.38:0/2684383177 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7b48011950 con 0x7f7b5407de10 2024-03-26T06:27:34.445 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.804+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/2684383177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b541a0780 con 0x7f7b5407de10 2024-03-26T06:27:34.445 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.804+0000 7f7b51ffb700 1 -- 172.21.15.38:0/2684383177 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f7b4801a460 con 0x7f7b5407de10 2024-03-26T06:27:34.445 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.804+0000 7f7b51ffb700 1 -- 172.21.15.38:0/2684383177 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7b4801b780 con 0x7f7b5407de10 2024-03-26T06:27:34.445 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.805+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/2684383177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b54068f20 con 0x7f7b5407de10 2024-03-26T06:27:34.445 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.807+0000 7f7b51ffb700 1 -- 172.21.15.38:0/2684383177 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f7b4801ba20 con 0x7f7b5407de10 2024-03-26T06:27:34.445 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.877+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/2684383177 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f7b5402db30 con 0x7f7b5407de10 2024-03-26T06:27:34.445 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.880+0000 7f7b51ffb700 1 -- 172.21.15.38:0/2684383177 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+454 (secure 0 0 0) 0x7f7b4801bc00 con 0x7f7b5407de10 2024-03-26T06:27:34.445 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.881+0000 7f7b51ffb700 1 -- 172.21.15.38:0/2684383177 <== mon.0 v2:172.21.15.38:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b4801bde0 con 0x7f7b5407de10 2024-03-26T06:27:34.445 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.881+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/2684383177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 msgr2=0x7f7b5419d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:34.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.881+0000 7f7b5aeb2700 1 --2- 172.21.15.38:0/2684383177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5419d310 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f7b48000c00 tx=0x7f7b48004750 comp rx=0 tx=0).stop 2024-03-26T06:27:34.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.881+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/2684383177 shutdown_connections 2024-03-26T06:27:34.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.881+0000 7f7b5aeb2700 1 --2- 172.21.15.38:0/2684383177 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7b5407de10 0x7f7b5419d310 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:34.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.881+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/2684383177 >> 172.21.15.38:0/2684383177 conn(0x7f7b54102fd0 msgr2=0x7f7b541052b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:34.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.881+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/2684383177 shutdown_connections 2024-03-26T06:27:34.446 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:33.881+0000 7f7b5aeb2700 1 -- 172.21.15.38:0/2684383177 wait complete. 2024-03-26T06:27:34.446 INFO:teuthology.orchestra.run.smithi038.stdout:Generating new minimal ceph.conf... 2024-03-26T06:27:34.558 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:34 smithi038 ceph-mon[31897]: from='client.? 172.21.15.38:0/2684383177' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T06:27:34.558 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:34 smithi038 ceph-mon[31897]: from='client.? 172.21.15.38:0/2684383177' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T06:27:35.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:35 smithi038 ceph-mon[31897]: from='client.? 172.21.15.38:0/2001199714' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:27:36.111 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.480+0000 7f3d0f4d4700 1 Processor -- start 2024-03-26T06:27:36.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.481+0000 7f3d0f4d4700 1 -- start start 2024-03-26T06:27:36.112 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.481+0000 7f3d0f4d4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d08106c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.481+0000 7f3d0f4d4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d08107250 con 0x7f3d08106860 2024-03-26T06:27:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.481+0000 7f3d0d270700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d08106c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.481+0000 7f3d0d270700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d08106c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37902/0 (socket says 172.21.15.38:37902) 2024-03-26T06:27:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.481+0000 7f3d0d270700 1 -- 172.21.15.38:0/2423152906 learned_addr learned my addr 172.21.15.38:0/2423152906 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.481+0000 7f3d0d270700 1 -- 172.21.15.38:0/2423152906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d08107a60 con 0x7f3d08106860 2024-03-26T06:27:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.481+0000 7f3d0d270700 1 --2- 172.21.15.38:0/2423152906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d08106c80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80098a0 tx=0x7f3cf8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a107650554f7c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.482+0000 7f3cfffff700 1 -- 172.21.15.38:0/2423152906 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cf8004160 con 0x7f3d08106860 2024-03-26T06:27:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.482+0000 7f3cfffff700 1 -- 172.21.15.38:0/2423152906 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cf8004300 con 0x7f3d08106860 2024-03-26T06:27:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.482+0000 7f3cfffff700 1 -- 172.21.15.38:0/2423152906 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cf8011b50 con 0x7f3d08106860 2024-03-26T06:27:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.482+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2423152906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 msgr2=0x7f3d08106c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:36.113 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.482+0000 7f3d0f4d4700 1 --2- 172.21.15.38:0/2423152906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d08106c80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f3cf80098a0 tx=0x7f3cf8009bb0 comp rx=0 tx=0).stop 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.482+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2423152906 shutdown_connections 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.482+0000 7f3d0f4d4700 1 --2- 172.21.15.38:0/2423152906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d08106c80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.482+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2423152906 >> 172.21.15.38:0/2423152906 conn(0x7f3d08101e40 msgr2=0x7f3d08104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.482+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2423152906 shutdown_connections 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.482+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2423152906 wait complete. 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.483+0000 7f3d0f4d4700 1 Processor -- start 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.483+0000 7f3d0f4d4700 1 -- start start 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.483+0000 7f3d0f4d4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d081919c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.483+0000 7f3d0f4d4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0810ae10 con 0x7f3d08106860 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.483+0000 7f3d0d270700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d081919c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.483+0000 7f3d0d270700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d081919c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37918/0 (socket says 172.21.15.38:37918) 2024-03-26T06:27:36.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.483+0000 7f3d0d270700 1 -- 172.21.15.38:0/2001199714 learned_addr learned my addr 172.21.15.38:0/2001199714 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.484+0000 7f3d0d270700 1 -- 172.21.15.38:0/2001199714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d08191f00 con 0x7f3d08106860 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.484+0000 7f3d0d270700 1 --2- 172.21.15.38:0/2001199714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d081919c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8009fb0 tx=0x7f3cf8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.484+0000 7f3cfe7fc700 1 -- 172.21.15.38:0/2001199714 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cf80038b0 con 0x7f3d08106860 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.484+0000 7f3cfe7fc700 1 -- 172.21.15.38:0/2001199714 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3cf8011e50 con 0x7f3d08106860 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.484+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2001199714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d08190140 con 0x7f3d08106860 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.484+0000 7f3cfe7fc700 1 -- 172.21.15.38:0/2001199714 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3cf8019550 con 0x7f3d08106860 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.484+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2001199714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d08190620 con 0x7f3d08106860 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.484+0000 7f3cfe7fc700 1 -- 172.21.15.38:0/2001199714 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f3cf8019be0 con 0x7f3d08106860 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.484+0000 7f3cfe7fc700 1 -- 172.21.15.38:0/2001199714 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f3cf8024730 con 0x7f3d08106860 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.485+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2001199714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d08068f20 con 0x7f3d08106860 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.487+0000 7f3cfe7fc700 1 -- 172.21.15.38:0/2001199714 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f3cf8003a50 con 0x7f3d08106860 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.558+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2001199714 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3d08195710 con 0x7f3d08106860 2024-03-26T06:27:36.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.558+0000 7f3cfe7fc700 1 -- 172.21.15.38:0/2001199714 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f3cf80249d0 con 0x7f3d08106860 2024-03-26T06:27:36.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.559+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2001199714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 msgr2=0x7f3d081919c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:36.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.559+0000 7f3d0f4d4700 1 --2- 172.21.15.38:0/2001199714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d081919c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3cf8009fb0 tx=0x7f3cf8011bf0 comp rx=0 tx=0).stop 2024-03-26T06:27:36.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.559+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2001199714 shutdown_connections 2024-03-26T06:27:36.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.559+0000 7f3d0f4d4700 1 --2- 172.21.15.38:0/2001199714 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d08106860 0x7f3d081919c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:36.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.559+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2001199714 >> 172.21.15.38:0/2001199714 conn(0x7f3d08101e40 msgr2=0x7f3d08102d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:36.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.559+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2001199714 shutdown_connections 2024-03-26T06:27:36.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:35.559+0000 7f3d0f4d4700 1 -- 172.21.15.38:0/2001199714 wait complete. 2024-03-26T06:27:36.116 INFO:teuthology.orchestra.run.smithi038.stdout:Restarting the monitor... 2024-03-26T06:27:36.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:36 smithi038 systemd[1]: Stopping Ceph mon.a for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:27:36.800 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:36 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a[31893]: 2024-03-26T06:27:36.364+0000 7f63135d7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:27:36.800 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:36 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a[31893]: 2024-03-26T06:27:36.364+0000 7f63135d7700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T06:27:36.800 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:36 smithi038 podman[32139]: 2024-03-26 06:27:36.53845631 +0000 UTC m=+0.268292456 container died f2223e52d85c9cce24eee274fb6ee7989085040fc9f22e4e14616808598042c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f) 2024-03-26T06:27:37.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:36 smithi038 podman[32139]: 2024-03-26 06:27:36.889471314 +0000 UTC m=+0.619307448 container cleanup f2223e52d85c9cce24eee274fb6ee7989085040fc9f22e4e14616808598042c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T06:27:37.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:36 smithi038 bash[32139]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a 2024-03-26T06:27:37.558 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 podman[32153]: 2024-03-26 06:27:37.198901955 +0000 UTC m=+0.664462680 container remove f2223e52d85c9cce24eee274fb6ee7989085040fc9f22e4e14616808598042c3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T06:27:37.559 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T06:27:37.559 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 systemd[1]: Stopped Ceph mon.a for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:27:37.559 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 systemd[1]: Starting Ceph mon.a for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:27:37.809 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 podman[32247]: 2024-03-26 06:27:37.557954942 +0000 UTC m=+0.101753989 container create 127b2bec7e81d0af976eb8c76ed4bf74de9bdc731e4ed441271857d57627d0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T06:27:37.809 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 podman[32247]: 2024-03-26 06:27:37.473261348 +0000 UTC m=+0.017060393 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:27:37.809 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 podman[32247]: 2024-03-26 06:27:37.77724367 +0000 UTC m=+0.321042715 container init 127b2bec7e81d0af976eb8c76ed4bf74de9bdc731e4ed441271857d57627d0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T06:27:37.809 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 podman[32247]: 2024-03-26 06:27:37.780210242 +0000 UTC m=+0.324009289 container start 127b2bec7e81d0af976eb8c76ed4bf74de9bdc731e4ed441271857d57627d0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f) 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: ceph version 19.0.0-1964-g7e98f5f6 (7e98f5f6b535defc7fa715b456904ffc0500aa9f) squid (dev), process ceph-mon, pid 2 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: pidfile_write: ignore empty --pid-file 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: load: jerasure load: lrc 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: RocksDB version: 7.9.2 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Git sha 0 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Compile date 2024-03-23 10:35:25 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: DB SUMMARY 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: DB Session ID: FZ4M5B2YRHIQA0BY2IJB 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: CURRENT file: CURRENT 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T06:27:37.810 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 101556 ; 2024-03-26T06:27:37.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.error_if_exists: 0 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.create_if_missing: 0 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.paranoid_checks: 1 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.env: 0x55820b29ab20 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.info_log: 0x55820d71f8e0 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.statistics: (nil) 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.use_fsync: 0 2024-03-26T06:27:37.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_log_file_size: 0 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.allow_fallocate: 1 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.use_direct_reads: 0 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.db_log_dir: 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.wal_dir: 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T06:27:37.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.write_buffer_manager: 0x55820d6f3400 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T06:27:37.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.unordered_write: 0 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.row_cache: None 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.wal_filter: None 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.two_write_queues: 0 2024-03-26T06:27:37.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.wal_compression: 0 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.atomic_flush: 0 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.log_readahead_size: 0 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_background_jobs: 2 2024-03-26T06:27:37.816 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_background_compactions: -1 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_subcompactions: 1 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_open_files: -1 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T06:27:37.817 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_background_flushes: -1 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Compression algorithms supported: 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: kZSTD supported: 0 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: kXpressCompression supported: 0 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: kZlibCompression supported: 1 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: kSnappyCompression supported: 1 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: kLZ4Compression supported: 1 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: kBZip2Compression supported: 0 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T06:27:37.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.merge_operator: 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_filter: None 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55820d71f9e0) 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks: 1 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_top_level_index_and_filter: 1 2024-03-26T06:27:37.819 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_type: 0 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_index_type: 0 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_shortening: 1 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: checksum: 4 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: no_block_cache: 0 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache: 0x55820d725090 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_name: BinnedLRUCache 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_options: 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: capacity : 536870912 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_shard_bits : 4 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: strict_capacity_limit : 0 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: high_pri_pool_ratio: 0.000 2024-03-26T06:27:37.820 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_compressed: (nil) 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: persistent_cache: (nil) 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size: 4096 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size_deviation: 10 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_restart_interval: 16 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_block_restart_interval: 1 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: metadata_block_size: 4096 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: partition_filters: 0 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: use_delta_encoding: 1 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: filter_policy: bloomfilter 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: whole_key_filtering: 1 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: verify_compression: 0 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: read_amp_bytes_per_bit: 0 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: format_version: 5 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_index_compression: 1 2024-03-26T06:27:37.821 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_align: 0 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout: max_auto_readahead_size: 262144 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout: prepopulate_block_cache: 0 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout: initial_auto_readahead_size: 8192 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compression: NoCompression 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.num_levels: 7 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T06:27:37.822 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T06:27:37.823 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T06:27:37.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T06:27:37.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T06:27:37.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T06:27:37.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T06:27:37.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T06:27:37.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T06:27:37.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T06:27:37.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T06:27:37.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T06:27:37.824 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T06:27:37.825 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T06:27:37.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T06:27:37.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T06:27:37.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T06:27:37.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T06:27:37.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T06:27:37.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T06:27:37.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T06:27:37.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T06:27:37.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T06:27:37.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T06:27:37.826 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T06:27:37.885 INFO:teuthology.orchestra.run.smithi038.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T06:27:38.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T06:27:38.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T06:27:38.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.table_properties_collectors: 2024-03-26T06:27:38.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.inplace_update_support: 0 2024-03-26T06:27:38.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T06:27:38.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T06:27:38.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.bloom_locality: 0 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.max_successive_merges: 0 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.ttl: 2592000 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.enable_blob_files: false 2024-03-26T06:27:38.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.min_blob_size: 0 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 19cc3122-99a4-4c9d-86d9-cf7687fee907 2024-03-26T06:27:38.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711434457808735, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711434457809665, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 96571, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 324, "table_properties": {"data_size": 94505, "index_size": 284, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14247, "raw_average_key_size": 50, "raw_value_size": 86810, "raw_average_value_size": 306, "num_data_blocks": 11, "num_entries": 283, "num_filter_entries": 283, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711434457, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "19cc3122-99a4-4c9d-86d9-cf7687fee907", "db_session_id": "FZ4M5B2YRHIQA0BY2IJB", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711434457809724, "job": 1, "event": "recovery_finished"} 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55820d818000 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: DB pointer 0x55820d802000 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** DB Stats ** 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T06:27:38.086 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: L0 2/0 96.14 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: Sum 2/0 96.14 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 166.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T06:27:38.087 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 166.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T06:27:38.088 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative compaction: 0.00 GB write, 28.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval compaction: 0.00 GB write, 28.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache BinnedLRUCache@0x55820d725090#2 capacity: 512.00 MB usage: 1.48 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: starting mon.a rank 0 at public addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] at bind addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: mon.a@-1(???) e1 preinit fsid c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: mon.a@-1(???).mds e1 new map 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: mon.a@-1(???).mds e1 print_map 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout: e1 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T06:27:38.089 INFO:journalctl@ceph.mon.a.smithi038.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout: legacy client fscid: -1 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout: No filesystems configured 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 bash[32247]: 127b2bec7e81d0af976eb8c76ed4bf74de9bdc731e4ed441271857d57627d0bb 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: monmap e1: 1 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: fsmap 2024-03-26T06:27:38.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T06:27:38.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 ceph-mon[32265]: mgrmap e1: no daemons active 2024-03-26T06:27:38.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:37 smithi038 systemd[1]: Started Ceph mon.a for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:27:38.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.283+0000 7fefb2a96700 1 Processor -- start 2024-03-26T06:27:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.284+0000 7fefb2a96700 1 -- start start 2024-03-26T06:27:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.284+0000 7fefb2a96700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac107e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.284+0000 7fefb2a96700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac108400 con 0x7fefac107a10 2024-03-26T06:27:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.284+0000 7fefb0832700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac107e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.284+0000 7fefb0832700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac107e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37932/0 (socket says 172.21.15.38:37932) 2024-03-26T06:27:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.284+0000 7fefb0832700 1 -- 172.21.15.38:0/2805186798 learned_addr learned my addr 172.21.15.38:0/2805186798 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.284+0000 7fefb0832700 1 -- 172.21.15.38:0/2805186798 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac108b70 con 0x7fefac107a10 2024-03-26T06:27:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.285+0000 7fefb0832700 1 --2- 172.21.15.38:0/2805186798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac107e30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fefa00098a0 tx=0x7fefa0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819b7ed408db75c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.285+0000 7fefab7fe700 1 -- 172.21.15.38:0/2805186798 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefa0004160 con 0x7fefac107a10 2024-03-26T06:27:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.285+0000 7fefab7fe700 1 -- 172.21.15.38:0/2805186798 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefa0004300 con 0x7fefac107a10 2024-03-26T06:27:38.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.285+0000 7fefab7fe700 1 -- 172.21.15.38:0/2805186798 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefa0011b50 con 0x7fefac107a10 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.285+0000 7fefb2a96700 1 -- 172.21.15.38:0/2805186798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 msgr2=0x7fefac107e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.285+0000 7fefb2a96700 1 --2- 172.21.15.38:0/2805186798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac107e30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fefa00098a0 tx=0x7fefa0009bb0 comp rx=0 tx=0).stop 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.285+0000 7fefb2a96700 1 -- 172.21.15.38:0/2805186798 shutdown_connections 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.285+0000 7fefb2a96700 1 --2- 172.21.15.38:0/2805186798 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac107e30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.285+0000 7fefb2a96700 1 -- 172.21.15.38:0/2805186798 >> 172.21.15.38:0/2805186798 conn(0x7fefac102fb0 msgr2=0x7fefac1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.285+0000 7fefb2a96700 1 -- 172.21.15.38:0/2805186798 shutdown_connections 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.286+0000 7fefb2a96700 1 -- 172.21.15.38:0/2805186798 wait complete. 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.286+0000 7fefb2a96700 1 Processor -- start 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.286+0000 7fefb2a96700 1 -- start start 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.286+0000 7fefb2a96700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac07daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.286+0000 7fefb2a96700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac10bf20 con 0x7fefac107a10 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.286+0000 7fefb0832700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac07daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:38.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.286+0000 7fefb0832700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac07daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37934/0 (socket says 172.21.15.38:37934) 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.286+0000 7fefb0832700 1 -- 172.21.15.38:0/3836289822 learned_addr learned my addr 172.21.15.38:0/3836289822 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.287+0000 7fefb0832700 1 -- 172.21.15.38:0/3836289822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac07e030 con 0x7fefac107a10 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.287+0000 7fefb0832700 1 --2- 172.21.15.38:0/3836289822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac07daf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fefa0000c00 tx=0x7fefa0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.287+0000 7fefa9ffb700 1 -- 172.21.15.38:0/3836289822 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefa0003b20 con 0x7fefac107a10 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.287+0000 7fefa9ffb700 1 -- 172.21.15.38:0/3836289822 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefa0003cc0 con 0x7fefac107a10 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.287+0000 7fefb2a96700 1 -- 172.21.15.38:0/3836289822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac07c220 con 0x7fefac107a10 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.287+0000 7fefa9ffb700 1 -- 172.21.15.38:0/3836289822 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fefa0024910 con 0x7fefac107a10 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.287+0000 7fefb2a96700 1 -- 172.21.15.38:0/3836289822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefac07c760 con 0x7fefac107a10 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.287+0000 7fefa9ffb700 1 -- 172.21.15.38:0/3836289822 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fefa0022020 con 0x7fefac107a10 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.287+0000 7fefa9ffb700 1 -- 172.21.15.38:0/3836289822 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fefa0011e00 con 0x7fefac107a10 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.288+0000 7fefb2a96700 1 -- 172.21.15.38:0/3836289822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefac063410 con 0x7fefac107a10 2024-03-26T06:27:38.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.291+0000 7fefa9ffb700 1 -- 172.21.15.38:0/3836289822 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7fefa002ddf0 con 0x7fefac107a10 2024-03-26T06:27:38.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.361+0000 7fefb2a96700 1 -- 172.21.15.38:0/3836289822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fefac07cc90 con 0x7fefac107a10 2024-03-26T06:27:38.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.363+0000 7fefa9ffb700 1 -- 172.21.15.38:0/3836289822 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fefa0032030 con 0x7fefac107a10 2024-03-26T06:27:38.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.364+0000 7fefb2a96700 1 -- 172.21.15.38:0/3836289822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 msgr2=0x7fefac07daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:38.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.364+0000 7fefb2a96700 1 --2- 172.21.15.38:0/3836289822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac07daf0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fefa0000c00 tx=0x7fefa0011c10 comp rx=0 tx=0).stop 2024-03-26T06:27:38.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.364+0000 7fefb2a96700 1 -- 172.21.15.38:0/3836289822 shutdown_connections 2024-03-26T06:27:38.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.364+0000 7fefb2a96700 1 --2- 172.21.15.38:0/3836289822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fefac107a10 0x7fefac07daf0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:38.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.364+0000 7fefb2a96700 1 -- 172.21.15.38:0/3836289822 >> 172.21.15.38:0/3836289822 conn(0x7fefac102fb0 msgr2=0x7fefac103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:38.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.364+0000 7fefb2a96700 1 -- 172.21.15.38:0/3836289822 shutdown_connections 2024-03-26T06:27:38.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:38.364+0000 7fefb2a96700 1 -- 172.21.15.38:0/3836289822 wait complete. 2024-03-26T06:27:38.879 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T06:27:38.892 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T06:27:38.892 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mgr... 2024-03-26T06:27:38.893 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T06:27:38.893 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T06:27:39.131 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.x 2024-03-26T06:27:39.131 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.x.service: Unit ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.x.service not loaded. 2024-03-26T06:27:39.277 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c2824556-eb39-11ee-95d0-87774f69a715.target.wants/ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-c2824556-eb39-11ee-95d0-87774f69a715@.service. 2024-03-26T06:27:39.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:39 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3836289822' entity='client.admin' 2024-03-26T06:27:39.756 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:39 smithi038 podman[32484]: 2024-03-26 06:27:39.690969279 +0000 UTC m=+0.300485502 container start 1c39329e522c80af668165827c481e4b7550c34923c3c70b44bbf0108a908f6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-26T06:27:39.756 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:39 smithi038 bash[32484]: 1c39329e522c80af668165827c481e4b7550c34923c3c70b44bbf0108a908f6a 2024-03-26T06:27:39.769 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T06:27:39.769 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-26T06:27:39.776 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T06:27:39.777 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T06:27:39.777 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-26T06:27:39.777 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T06:27:39.787 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T06:27:39.787 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-26T06:27:39.795 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T06:27:39.795 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T06:27:39.795 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-26T06:27:39.795 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T06:27:39.795 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr to start... 2024-03-26T06:27:39.795 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr... 2024-03-26T06:27:40.038 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:39 smithi038 systemd[1]: Started Ceph mgr.x for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:27:40.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:40 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1727112785' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T06:27:40.582 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:40 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:40.287+0000 7f327336b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T06:27:40.583 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:40 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:40.390+0000 7f327336b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T06:27:40.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:40.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T06:27:40.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "c2824556-eb39-11ee-95d0-87774f69a715", 2024-03-26T06:27:40.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T06:27:40.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T06:27:40.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T06:27:40.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T06:27:40.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:40.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T06:27:40.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T06:27:40.879 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T06:27:40.880 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:40.881 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T06:27:40.881 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T06:27:40.881 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T06:27:40.881 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T06:27:40.881 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T06:27:40.881 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T06:27:40.891 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T06:27:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T06:27:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T06:27:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T06:27:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T06:27:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T06:27:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T06:27:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T06:27:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T06:27:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:40.892 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T06:27:40.893 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:40.893 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T06:27:40.893 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T06:27:40.893 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:40.893 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T06:27:40.893 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T06:27:40.893 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T06:27:40.893 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T06:27:40.893 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T06:27:40.893 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T06:27:40.893 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T06:27:40.894 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:40.894 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T06:27:40.894 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:40.894 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T06:27:40.894 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:40.894 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T06:27:32.376835+0000", 2024-03-26T06:27:40.894 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T06:27:40.894 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:40.894 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T06:27:40.894 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T06:27:40.894 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.242+0000 7f9cd24dd700 1 Processor -- start 2024-03-26T06:27:40.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.242+0000 7f9cd24dd700 1 -- start start 2024-03-26T06:27:40.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.242+0000 7f9cd24dd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc106cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:40.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.242+0000 7f9cd24dd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ccc107280 con 0x7f9ccc106890 2024-03-26T06:27:40.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.242+0000 7f9ccbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc106cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:40.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.242+0000 7f9ccbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc106cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37954/0 (socket says 172.21.15.38:37954) 2024-03-26T06:27:40.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.242+0000 7f9ccbfff700 1 -- 172.21.15.38:0/3511396962 learned_addr learned my addr 172.21.15.38:0/3511396962 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:40.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.242+0000 7f9ccbfff700 1 -- 172.21.15.38:0/3511396962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc107a90 con 0x7f9ccc106890 2024-03-26T06:27:40.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.243+0000 7f9ccbfff700 1 --2- 172.21.15.38:0/3511396962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc106cb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0098a0 tx=0x7f9cbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cac56769a115978 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:40.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.243+0000 7f9ccaffd700 1 -- 172.21.15.38:0/3511396962 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cbc004160 con 0x7f9ccc106890 2024-03-26T06:27:40.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.243+0000 7f9ccaffd700 1 -- 172.21.15.38:0/3511396962 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cbc004300 con 0x7f9ccc106890 2024-03-26T06:27:40.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.243+0000 7f9ccaffd700 1 -- 172.21.15.38:0/3511396962 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cbc011b50 con 0x7f9ccc106890 2024-03-26T06:27:40.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.244+0000 7f9cd24dd700 1 -- 172.21.15.38:0/3511396962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 msgr2=0x7f9ccc106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:40.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.244+0000 7f9cd24dd700 1 --2- 172.21.15.38:0/3511396962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc106cb0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc0098a0 tx=0x7f9cbc009bb0 comp rx=0 tx=0).stop 2024-03-26T06:27:40.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.244+0000 7f9cd24dd700 1 -- 172.21.15.38:0/3511396962 shutdown_connections 2024-03-26T06:27:40.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.244+0000 7f9cd24dd700 1 --2- 172.21.15.38:0/3511396962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc106cb0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:40.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.244+0000 7f9cd24dd700 1 -- 172.21.15.38:0/3511396962 >> 172.21.15.38:0/3511396962 conn(0x7f9ccc101e50 msgr2=0x7f9ccc104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:40.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.244+0000 7f9cd24dd700 1 -- 172.21.15.38:0/3511396962 shutdown_connections 2024-03-26T06:27:40.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.244+0000 7f9cd24dd700 1 -- 172.21.15.38:0/3511396962 wait complete. 2024-03-26T06:27:40.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.244+0000 7f9cd24dd700 1 Processor -- start 2024-03-26T06:27:40.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.244+0000 7f9cd24dd700 1 -- start start 2024-03-26T06:27:40.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.245+0000 7f9cd24dd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc191a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:40.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.245+0000 7f9cd24dd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ccc10ae40 con 0x7f9ccc106890 2024-03-26T06:27:40.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.245+0000 7f9ccbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc191a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:40.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.245+0000 7f9ccbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc191a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37966/0 (socket says 172.21.15.38:37966) 2024-03-26T06:27:40.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.245+0000 7f9ccbfff700 1 -- 172.21.15.38:0/1727112785 learned_addr learned my addr 172.21.15.38:0/1727112785 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:40.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.245+0000 7f9ccbfff700 1 -- 172.21.15.38:0/1727112785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ccc191f80 con 0x7f9ccc106890 2024-03-26T06:27:40.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.245+0000 7f9ccbfff700 1 --2- 172.21.15.38:0/1727112785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc191a40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc009fb0 tx=0x7f9cbc011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:40.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.245+0000 7f9cc97fa700 1 -- 172.21.15.38:0/1727112785 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cbc0038b0 con 0x7f9ccc106890 2024-03-26T06:27:40.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.245+0000 7f9cc97fa700 1 -- 172.21.15.38:0/1727112785 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9cbc011e50 con 0x7f9ccc106890 2024-03-26T06:27:40.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.245+0000 7f9cd24dd700 1 -- 172.21.15.38:0/1727112785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ccc190170 con 0x7f9ccc106890 2024-03-26T06:27:40.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.246+0000 7f9cc97fa700 1 -- 172.21.15.38:0/1727112785 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9cbc019550 con 0x7f9ccc106890 2024-03-26T06:27:40.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.246+0000 7f9cd24dd700 1 -- 172.21.15.38:0/1727112785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ccc1906b0 con 0x7f9ccc106890 2024-03-26T06:27:40.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.246+0000 7f9cc97fa700 1 -- 172.21.15.38:0/1727112785 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9cbc019be0 con 0x7f9ccc106890 2024-03-26T06:27:40.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.246+0000 7f9cc97fa700 1 -- 172.21.15.38:0/1727112785 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9cbc024730 con 0x7f9ccc106890 2024-03-26T06:27:40.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.246+0000 7f9cd24dd700 1 -- 172.21.15.38:0/1727112785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ccc068f20 con 0x7f9ccc106890 2024-03-26T06:27:40.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.249+0000 7f9cc97fa700 1 -- 172.21.15.38:0/1727112785 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f9cbc003a50 con 0x7f9ccc106890 2024-03-26T06:27:40.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.348+0000 7f9cd24dd700 1 -- 172.21.15.38:0/1727112785 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9ccc195350 con 0x7f9ccc106890 2024-03-26T06:27:40.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.348+0000 7f9cc97fa700 1 -- 172.21.15.38:0/1727112785 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f9cbc024930 con 0x7f9ccc106890 2024-03-26T06:27:40.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.349+0000 7f9cd24dd700 1 -- 172.21.15.38:0/1727112785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 msgr2=0x7f9ccc191a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:40.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.349+0000 7f9cd24dd700 1 --2- 172.21.15.38:0/1727112785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc191a40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc009fb0 tx=0x7f9cbc011bf0 comp rx=0 tx=0).stop 2024-03-26T06:27:40.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.349+0000 7f9cd24dd700 1 -- 172.21.15.38:0/1727112785 shutdown_connections 2024-03-26T06:27:40.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.349+0000 7f9cd24dd700 1 --2- 172.21.15.38:0/1727112785 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ccc106890 0x7f9ccc191a40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:40.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.349+0000 7f9cd24dd700 1 -- 172.21.15.38:0/1727112785 >> 172.21.15.38:0/1727112785 conn(0x7f9ccc101e50 msgr2=0x7f9ccc102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:40.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.349+0000 7f9cd24dd700 1 -- 172.21.15.38:0/1727112785 shutdown_connections 2024-03-26T06:27:40.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:40.349+0000 7f9cd24dd700 1 -- 172.21.15.38:0/1727112785 wait complete. 2024-03-26T06:27:40.899 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (1/15)... 2024-03-26T06:27:41.157 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:41 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:41.062+0000 7f327336b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T06:27:41.157 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:41 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:41.155+0000 7f327336b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T06:27:41.482 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:41 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:41.298+0000 7f327336b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T06:27:41.482 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:41 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:41.395+0000 7f327336b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T06:27:41.482 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:41 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:41.480+0000 7f327336b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T06:27:42.043 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:41 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:41.747+0000 7f327336b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T06:27:42.043 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:41 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:41.858+0000 7f327336b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T06:27:42.043 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:41 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:41.945+0000 7f327336b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T06:27:42.043 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:42 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:42.041+0000 7f327336b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T06:27:42.294 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:42 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:42.219+0000 7f327336b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T06:27:43.266 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:43 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:43.234+0000 7f327336b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T06:27:43.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:43 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1518214113' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T06:27:43.583 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:43 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:43.371+0000 7f327336b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T06:27:43.991 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:43 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:43.711+0000 7f327336b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T06:27:43.991 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:43 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:43.804+0000 7f327336b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T06:27:44.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:44.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T06:27:44.115 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "c2824556-eb39-11ee-95d0-87774f69a715", 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:44.116 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T06:27:44.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T06:27:44.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:44.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T06:27:44.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T06:27:44.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T06:27:44.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T06:27:44.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T06:27:44.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T06:27:44.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T06:27:44.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T06:27:44.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T06:27:44.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T06:27:44.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T06:27:32.376835+0000", 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.280+0000 7fd766fe8700 1 Processor -- start 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.280+0000 7fd766fe8700 1 -- start start 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.280+0000 7fd766fe8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76007c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:44.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.280+0000 7fd766fe8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76007c850 con 0x7fd76007de20 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.281+0000 7fd764d84700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76007c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.281+0000 7fd764d84700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76007c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37974/0 (socket says 172.21.15.38:37974) 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.281+0000 7fd764d84700 1 -- 172.21.15.38:0/2090021614 learned_addr learned my addr 172.21.15.38:0/2090021614 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.281+0000 7fd764d84700 1 -- 172.21.15.38:0/2090021614 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76007c9d0 con 0x7fd76007de20 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.281+0000 7fd764d84700 1 --2- 172.21.15.38:0/2090021614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76007c280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd7540098a0 tx=0x7fd754009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1a7d8d01e834c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.281+0000 7fd75f7fe700 1 -- 172.21.15.38:0/2090021614 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd754004160 con 0x7fd76007de20 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.281+0000 7fd75f7fe700 1 -- 172.21.15.38:0/2090021614 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd754004300 con 0x7fd76007de20 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.281+0000 7fd75f7fe700 1 -- 172.21.15.38:0/2090021614 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd754011b50 con 0x7fd76007de20 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.282+0000 7fd766fe8700 1 -- 172.21.15.38:0/2090021614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 msgr2=0x7fd76007c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.282+0000 7fd766fe8700 1 --2- 172.21.15.38:0/2090021614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76007c280 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd7540098a0 tx=0x7fd754009bb0 comp rx=0 tx=0).stop 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.282+0000 7fd766fe8700 1 -- 172.21.15.38:0/2090021614 shutdown_connections 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.282+0000 7fd766fe8700 1 --2- 172.21.15.38:0/2090021614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76007c280 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:44.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.282+0000 7fd766fe8700 1 -- 172.21.15.38:0/2090021614 >> 172.21.15.38:0/2090021614 conn(0x7fd760102f60 msgr2=0x7fd760105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.282+0000 7fd766fe8700 1 -- 172.21.15.38:0/2090021614 shutdown_connections 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.282+0000 7fd766fe8700 1 -- 172.21.15.38:0/2090021614 wait complete. 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.283+0000 7fd766fe8700 1 Processor -- start 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.283+0000 7fd766fe8700 1 -- start start 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.283+0000 7fd766fe8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76019db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.283+0000 7fd766fe8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76010df70 con 0x7fd76007de20 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.283+0000 7fd764d84700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76019db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.283+0000 7fd764d84700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76019db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37982/0 (socket says 172.21.15.38:37982) 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.283+0000 7fd764d84700 1 -- 172.21.15.38:0/1518214113 learned_addr learned my addr 172.21.15.38:0/1518214113 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.283+0000 7fd764d84700 1 -- 172.21.15.38:0/1518214113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76019e070 con 0x7fd76007de20 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.283+0000 7fd764d84700 1 --2- 172.21.15.38:0/1518214113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76019db30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd754000c00 tx=0x7fd754011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.284+0000 7fd75dffb700 1 -- 172.21.15.38:0/1518214113 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd754003b20 con 0x7fd76007de20 2024-03-26T06:27:44.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.284+0000 7fd75dffb700 1 -- 172.21.15.38:0/1518214113 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd754003cc0 con 0x7fd76007de20 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.284+0000 7fd766fe8700 1 -- 172.21.15.38:0/1518214113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76019e300 con 0x7fd76007de20 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.284+0000 7fd75dffb700 1 -- 172.21.15.38:0/1518214113 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd754024910 con 0x7fd76007de20 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.284+0000 7fd766fe8700 1 -- 172.21.15.38:0/1518214113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7601a0ff0 con 0x7fd76007de20 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.284+0000 7fd75dffb700 1 -- 172.21.15.38:0/1518214113 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fd754022020 con 0x7fd76007de20 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.284+0000 7fd75dffb700 1 -- 172.21.15.38:0/1518214113 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd754011e00 con 0x7fd76007de20 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.285+0000 7fd766fe8700 1 -- 172.21.15.38:0/1518214113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd760068f20 con 0x7fd76007de20 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.287+0000 7fd75dffb700 1 -- 172.21.15.38:0/1518214113 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7fd75401acd0 con 0x7fd76007de20 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.360+0000 7fd766fe8700 1 -- 172.21.15.38:0/1518214113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd7601a11c0 con 0x7fd76007de20 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.360+0000 7fd75dffb700 1 -- 172.21.15.38:0/1518214113 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fd754032030 con 0x7fd76007de20 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.361+0000 7fd766fe8700 1 -- 172.21.15.38:0/1518214113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 msgr2=0x7fd76019db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.361+0000 7fd766fe8700 1 --2- 172.21.15.38:0/1518214113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76019db30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd754000c00 tx=0x7fd754011c10 comp rx=0 tx=0).stop 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.361+0000 7fd766fe8700 1 -- 172.21.15.38:0/1518214113 shutdown_connections 2024-03-26T06:27:44.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.362+0000 7fd766fe8700 1 --2- 172.21.15.38:0/1518214113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd76007de20 0x7fd76019db30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:44.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.362+0000 7fd766fe8700 1 -- 172.21.15.38:0/1518214113 >> 172.21.15.38:0/1518214113 conn(0x7fd760102f60 msgr2=0x7fd760103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:44.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.362+0000 7fd766fe8700 1 -- 172.21.15.38:0/1518214113 shutdown_connections 2024-03-26T06:27:44.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:43.362+0000 7fd766fe8700 1 -- 172.21.15.38:0/1518214113 wait complete. 2024-03-26T06:27:44.124 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (2/15)... 2024-03-26T06:27:44.332 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:44 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:44.003+0000 7f327336b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:27:44.332 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:44 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:44.193+0000 7f327336b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T06:27:44.748 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:44 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:44.485+0000 7f327336b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:27:44.749 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:44 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:44.746+0000 7f327336b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T06:27:45.082 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:44 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:44.989+0000 7f327336b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T06:27:46.082 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:45 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:45.821+0000 7f327336b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T06:27:46.506 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:46 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:46.408+0000 7f327336b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T06:27:46.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:46 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3994924830' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T06:27:46.833 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:46 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:46.512+0000 7f327336b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T06:27:46.833 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:46 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:46.746+0000 7f327336b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T06:27:47.131 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:47.131 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T06:27:47.131 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "c2824556-eb39-11ee-95d0-87774f69a715", 2024-03-26T06:27:47.131 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T06:27:47.131 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T06:27:47.131 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T06:27:47.131 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T06:27:47.131 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:47.131 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:47.132 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T06:27:47.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:47.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T06:27:47.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T06:27:47.133 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T06:27:47.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T06:27:47.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T06:27:47.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T06:27:47.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:47.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T06:27:47.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T06:27:47.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T06:27:47.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T06:27:47.142 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T06:27:47.143 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T06:27:47.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T06:27:47.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T06:27:47.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T06:27:47.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T06:27:47.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:47.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T06:27:47.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:47.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T06:27:47.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:47.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T06:27:32.376835+0000", 2024-03-26T06:27:47.144 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T06:27:47.145 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:47.145 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T06:27:47.145 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T06:27:47.145 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.511+0000 7fb3805d3700 1 Processor -- start 2024-03-26T06:27:47.145 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.511+0000 7fb3805d3700 1 -- start start 2024-03-26T06:27:47.145 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.511+0000 7fb3805d3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb378106690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:47.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.511+0000 7fb3805d3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb37807bdb0 con 0x7fb378106270 2024-03-26T06:27:47.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.511+0000 7fb37e36f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb378106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:47.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.511+0000 7fb37e36f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb378106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33168/0 (socket says 172.21.15.38:33168) 2024-03-26T06:27:47.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.511+0000 7fb37e36f700 1 -- 172.21.15.38:0/1881156074 learned_addr learned my addr 172.21.15.38:0/1881156074 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:47.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.512+0000 7fb37e36f700 1 -- 172.21.15.38:0/1881156074 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb378106bd0 con 0x7fb378106270 2024-03-26T06:27:47.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.512+0000 7fb37e36f700 1 --2- 172.21.15.38:0/1881156074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb378106690 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0098a0 tx=0x7fb36c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6b0db53af5dd8cf server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:47.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.512+0000 7fb37d36d700 1 -- 172.21.15.38:0/1881156074 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb36c004160 con 0x7fb378106270 2024-03-26T06:27:47.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.512+0000 7fb37d36d700 1 -- 172.21.15.38:0/1881156074 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb36c004300 con 0x7fb378106270 2024-03-26T06:27:47.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.512+0000 7fb37d36d700 1 -- 172.21.15.38:0/1881156074 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb36c011b50 con 0x7fb378106270 2024-03-26T06:27:47.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.513+0000 7fb3805d3700 1 -- 172.21.15.38:0/1881156074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 msgr2=0x7fb378106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:47.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.513+0000 7fb3805d3700 1 --2- 172.21.15.38:0/1881156074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb378106690 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0098a0 tx=0x7fb36c009bb0 comp rx=0 tx=0).stop 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.513+0000 7fb3805d3700 1 -- 172.21.15.38:0/1881156074 shutdown_connections 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.513+0000 7fb3805d3700 1 --2- 172.21.15.38:0/1881156074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb378106690 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.513+0000 7fb3805d3700 1 -- 172.21.15.38:0/1881156074 >> 172.21.15.38:0/1881156074 conn(0x7fb378101e10 msgr2=0x7fb378104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.513+0000 7fb3805d3700 1 -- 172.21.15.38:0/1881156074 shutdown_connections 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.513+0000 7fb3805d3700 1 -- 172.21.15.38:0/1881156074 wait complete. 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.514+0000 7fb3805d3700 1 Processor -- start 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.514+0000 7fb3805d3700 1 -- start start 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.514+0000 7fb3805d3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb37819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.514+0000 7fb3805d3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb378107c60 con 0x7fb378106270 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.514+0000 7fb37e36f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb37819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.514+0000 7fb37e36f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb37819db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33176/0 (socket says 172.21.15.38:33176) 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.514+0000 7fb37e36f700 1 -- 172.21.15.38:0/3994924830 learned_addr learned my addr 172.21.15.38:0/3994924830 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:47.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.514+0000 7fb37e36f700 1 -- 172.21.15.38:0/3994924830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37819e0a0 con 0x7fb378106270 2024-03-26T06:27:47.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.514+0000 7fb37e36f700 1 --2- 172.21.15.38:0/3994924830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb37819db60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb36c009fb0 tx=0x7fb36c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:47.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.514+0000 7fb36b7fe700 1 -- 172.21.15.38:0/3994924830 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb36c0038b0 con 0x7fb378106270 2024-03-26T06:27:47.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.514+0000 7fb36b7fe700 1 -- 172.21.15.38:0/3994924830 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb36c011e50 con 0x7fb378106270 2024-03-26T06:27:47.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.515+0000 7fb3805d3700 1 -- 172.21.15.38:0/3994924830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb37819e330 con 0x7fb378106270 2024-03-26T06:27:47.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.515+0000 7fb36b7fe700 1 -- 172.21.15.38:0/3994924830 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb36c019420 con 0x7fb378106270 2024-03-26T06:27:47.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.515+0000 7fb3805d3700 1 -- 172.21.15.38:0/3994924830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3781a1020 con 0x7fb378106270 2024-03-26T06:27:47.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.515+0000 7fb36b7fe700 1 -- 172.21.15.38:0/3994924830 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb36c019a80 con 0x7fb378106270 2024-03-26T06:27:47.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.515+0000 7fb36b7fe700 1 -- 172.21.15.38:0/3994924830 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb36c024930 con 0x7fb378106270 2024-03-26T06:27:47.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.516+0000 7fb3805d3700 1 -- 172.21.15.38:0/3994924830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb378068f20 con 0x7fb378106270 2024-03-26T06:27:47.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.518+0000 7fb36b7fe700 1 -- 172.21.15.38:0/3994924830 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7fb36c043e50 con 0x7fb378106270 2024-03-26T06:27:47.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.593+0000 7fb3805d3700 1 -- 172.21.15.38:0/3994924830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb37802da30 con 0x7fb378106270 2024-03-26T06:27:47.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.593+0000 7fb36b7fe700 1 -- 172.21.15.38:0/3994924830 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fb36c032020 con 0x7fb378106270 2024-03-26T06:27:47.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.594+0000 7fb3805d3700 1 -- 172.21.15.38:0/3994924830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 msgr2=0x7fb37819db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:47.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.594+0000 7fb3805d3700 1 --2- 172.21.15.38:0/3994924830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb37819db60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb36c009fb0 tx=0x7fb36c011bf0 comp rx=0 tx=0).stop 2024-03-26T06:27:47.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.594+0000 7fb3805d3700 1 -- 172.21.15.38:0/3994924830 shutdown_connections 2024-03-26T06:27:47.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.594+0000 7fb3805d3700 1 --2- 172.21.15.38:0/3994924830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb378106270 0x7fb37819db60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:47.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.594+0000 7fb3805d3700 1 -- 172.21.15.38:0/3994924830 >> 172.21.15.38:0/3994924830 conn(0x7fb378101e10 msgr2=0x7fb3781947d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:47.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.595+0000 7fb3805d3700 1 -- 172.21.15.38:0/3994924830 shutdown_connections 2024-03-26T06:27:47.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:46.595+0000 7fb3805d3700 1 -- 172.21.15.38:0/3994924830 wait complete. 2024-03-26T06:27:47.149 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (3/15)... 2024-03-26T06:27:48.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: Activating manager daemon x 2024-03-26T06:27:48.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: mgrmap e2: x(active, starting, since 0.00327252s) 2024-03-26T06:27:48.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: from='mgr.14100 172.21.15.38:0/1122535081' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T06:27:48.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: from='mgr.14100 172.21.15.38:0/1122535081' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T06:27:48.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: from='mgr.14100 172.21.15.38:0/1122535081' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T06:27:48.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: from='mgr.14100 172.21.15.38:0/1122535081' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:27:48.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: from='mgr.14100 172.21.15.38:0/1122535081' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T06:27:48.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: Manager daemon x is now available 2024-03-26T06:27:48.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: from='mgr.14100 172.21.15.38:0/1122535081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T06:27:48.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: from='mgr.14100 172.21.15.38:0/1122535081' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T06:27:48.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: from='mgr.14100 172.21.15.38:0/1122535081' entity='mgr.x' 2024-03-26T06:27:48.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: from='mgr.14100 172.21.15.38:0/1122535081' entity='mgr.x' 2024-03-26T06:27:48.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:47 smithi038 ceph-mon[32265]: from='mgr.14100 172.21.15.38:0/1122535081' entity='mgr.x' 2024-03-26T06:27:49.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:48 smithi038 ceph-mon[32265]: mgrmap e3: x(active, since 1.00681s) 2024-03-26T06:27:50.413 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "c2824556-eb39-11ee-95d0-87774f69a715", 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:50.414 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T06:27:50.415 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T06:27:50.424 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T06:27:50.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T06:27:50.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T06:27:50.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:50.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T06:27:50.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T06:27:50.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T06:27:50.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T06:27:50.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T06:27:50.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T06:27:50.425 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T06:27:50.426 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T06:27:32.376835+0000", 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T06:27:50.427 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.568+0000 7fe413041700 1 Processor -- start 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.568+0000 7fe413041700 1 -- start start 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.569+0000 7fe413041700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c106cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.569+0000 7fe413041700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c107280 con 0x7fe40c106890 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.569+0000 7fe410ddd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c106cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.569+0000 7fe410ddd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c106cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33232/0 (socket says 172.21.15.38:33232) 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.569+0000 7fe410ddd700 1 -- 172.21.15.38:0/1833272820 learned_addr learned my addr 172.21.15.38:0/1833272820 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.569+0000 7fe410ddd700 1 -- 172.21.15.38:0/1833272820 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c107a90 con 0x7fe40c106890 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.569+0000 7fe410ddd700 1 --2- 172.21.15.38:0/1833272820 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c106cb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0098a0 tx=0x7fe3fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3807ebcb3ff40819 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.569+0000 7fe40b7fe700 1 -- 172.21.15.38:0/1833272820 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3fc004160 con 0x7fe40c106890 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.569+0000 7fe40b7fe700 1 -- 172.21.15.38:0/1833272820 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3fc004300 con 0x7fe40c106890 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.569+0000 7fe40b7fe700 1 -- 172.21.15.38:0/1833272820 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3fc011b50 con 0x7fe40c106890 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.570+0000 7fe413041700 1 -- 172.21.15.38:0/1833272820 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 msgr2=0x7fe40c106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:50.428 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.570+0000 7fe413041700 1 --2- 172.21.15.38:0/1833272820 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c106cb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0098a0 tx=0x7fe3fc009bb0 comp rx=0 tx=0).stop 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.570+0000 7fe413041700 1 -- 172.21.15.38:0/1833272820 shutdown_connections 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.570+0000 7fe413041700 1 --2- 172.21.15.38:0/1833272820 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c106cb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.570+0000 7fe413041700 1 -- 172.21.15.38:0/1833272820 >> 172.21.15.38:0/1833272820 conn(0x7fe40c101e10 msgr2=0x7fe40c104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.570+0000 7fe413041700 1 -- 172.21.15.38:0/1833272820 shutdown_connections 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.570+0000 7fe413041700 1 -- 172.21.15.38:0/1833272820 wait complete. 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.571+0000 7fe413041700 1 Processor -- start 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.571+0000 7fe413041700 1 -- start start 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.571+0000 7fe413041700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c19b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.571+0000 7fe413041700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c10ae40 con 0x7fe40c106890 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.571+0000 7fe410ddd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c19b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.571+0000 7fe410ddd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c19b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33240/0 (socket says 172.21.15.38:33240) 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.571+0000 7fe410ddd700 1 -- 172.21.15.38:0/3742437810 learned_addr learned my addr 172.21.15.38:0/3742437810 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.571+0000 7fe410ddd700 1 -- 172.21.15.38:0/3742437810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c19bf30 con 0x7fe40c106890 2024-03-26T06:27:50.429 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.572+0000 7fe410ddd700 1 --2- 172.21.15.38:0/3742437810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c19b9f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009fb0 tx=0x7fe3fc011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.572+0000 7fe409ffb700 1 -- 172.21.15.38:0/3742437810 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3fc0038b0 con 0x7fe40c106890 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.572+0000 7fe409ffb700 1 -- 172.21.15.38:0/3742437810 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe3fc011e50 con 0x7fe40c106890 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.572+0000 7fe409ffb700 1 -- 172.21.15.38:0/3742437810 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe3fc019550 con 0x7fe40c106890 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.572+0000 7fe413041700 1 -- 172.21.15.38:0/3742437810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c19c1c0 con 0x7fe40c106890 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.572+0000 7fe413041700 1 -- 172.21.15.38:0/3742437810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c19eeb0 con 0x7fe40c106890 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.573+0000 7fe409ffb700 1 -- 172.21.15.38:0/3742437810 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 3) v1 ==== 45989+0+0 (secure 0 0 0) 0x7fe3fc003a50 con 0x7fe40c106890 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.573+0000 7fe413041700 1 -- 172.21.15.38:0/3742437810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c068f20 con 0x7fe40c106890 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.573+0000 7fe409ffb700 1 --2- 172.21.15.38:0/3742437810 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fe3f4038dc0 0x7fe3f403b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.573+0000 7fe409ffb700 1 -- 172.21.15.38:0/3742437810 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe3fc04ce60 con 0x7fe40c106890 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.573+0000 7fe40bfff700 1 --2- 172.21.15.38:0/3742437810 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fe3f4038dc0 0x7fe3f403b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.574+0000 7fe40bfff700 1 --2- 172.21.15.38:0/3742437810 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fe3f4038dc0 0x7fe3f403b280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe400006fd0 tx=0x7fe400006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:50.430 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.579+0000 7fe409ffb700 1 -- 172.21.15.38:0/3742437810 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe3fc0198e0 con 0x7fe40c106890 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.769+0000 7fe409ffb700 1 -- 172.21.15.38:0/3742437810 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7fe3fc023390 con 0x7fe40c106890 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.865+0000 7fe413041700 1 -- 172.21.15.38:0/3742437810 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fe40c19f4f0 con 0x7fe40c106890 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.865+0000 7fe409ffb700 1 -- 172.21.15.38:0/3742437810 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fe3fc044a80 con 0x7fe40c106890 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.868+0000 7fe413041700 1 -- 172.21.15.38:0/3742437810 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fe3f4038dc0 msgr2=0x7fe3f403b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.868+0000 7fe413041700 1 --2- 172.21.15.38:0/3742437810 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fe3f4038dc0 0x7fe3f403b280 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe400006fd0 tx=0x7fe400006e40 comp rx=0 tx=0).stop 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.868+0000 7fe413041700 1 -- 172.21.15.38:0/3742437810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 msgr2=0x7fe40c19b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.868+0000 7fe413041700 1 --2- 172.21.15.38:0/3742437810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c19b9f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009fb0 tx=0x7fe3fc011bf0 comp rx=0 tx=0).stop 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.868+0000 7fe413041700 1 -- 172.21.15.38:0/3742437810 shutdown_connections 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.868+0000 7fe413041700 1 --2- 172.21.15.38:0/3742437810 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fe3f4038dc0 0x7fe3f403b280 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.868+0000 7fe413041700 1 --2- 172.21.15.38:0/3742437810 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe40c106890 0x7fe40c19b9f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.868+0000 7fe413041700 1 -- 172.21.15.38:0/3742437810 >> 172.21.15.38:0/3742437810 conn(0x7fe40c101e10 msgr2=0x7fe40c102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.868+0000 7fe413041700 1 -- 172.21.15.38:0/3742437810 shutdown_connections 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:49.868+0000 7fe413041700 1 -- 172.21.15.38:0/3742437810 wait complete. 2024-03-26T06:27:50.431 INFO:teuthology.orchestra.run.smithi038.stdout:mgr is available 2024-03-26T06:27:50.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:50 smithi038 ceph-mon[32265]: mgrmap e4: x(active, since 3s) 2024-03-26T06:27:50.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:50 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3742437810' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T06:27:51.600 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T06:27:51.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.852+0000 7f79e138e700 1 Processor -- start 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.853+0000 7f79e138e700 1 -- start start 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.853+0000 7f79e138e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc107e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.853+0000 7f79e138e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc1083e0 con 0x7f79dc1079f0 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.853+0000 7f79daffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc107e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.853+0000 7f79daffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc107e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33242/0 (socket says 172.21.15.38:33242) 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.853+0000 7f79daffd700 1 -- 172.21.15.38:0/4032381971 learned_addr learned my addr 172.21.15.38:0/4032381971 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:51.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.853+0000 7f79daffd700 1 -- 172.21.15.38:0/4032381971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc108ba0 con 0x7f79dc1079f0 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.853+0000 7f79daffd700 1 --2- 172.21.15.38:0/4032381971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc107e10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0098a0 tx=0x7f79cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23b96e2d36fb87f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.853+0000 7f79d9ffb700 1 -- 172.21.15.38:0/4032381971 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79cc004160 con 0x7f79dc1079f0 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.854+0000 7f79d9ffb700 1 -- 172.21.15.38:0/4032381971 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79cc004300 con 0x7f79dc1079f0 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.854+0000 7f79d9ffb700 1 -- 172.21.15.38:0/4032381971 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79cc011b50 con 0x7f79dc1079f0 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.854+0000 7f79e138e700 1 -- 172.21.15.38:0/4032381971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 msgr2=0x7f79dc107e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.854+0000 7f79e138e700 1 --2- 172.21.15.38:0/4032381971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc107e10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f79cc0098a0 tx=0x7f79cc009bb0 comp rx=0 tx=0).stop 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.854+0000 7f79e138e700 1 -- 172.21.15.38:0/4032381971 shutdown_connections 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.854+0000 7f79e138e700 1 --2- 172.21.15.38:0/4032381971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc107e10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.854+0000 7f79e138e700 1 -- 172.21.15.38:0/4032381971 >> 172.21.15.38:0/4032381971 conn(0x7f79dc102fd0 msgr2=0x7f79dc1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.854+0000 7f79e138e700 1 -- 172.21.15.38:0/4032381971 shutdown_connections 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.854+0000 7f79e138e700 1 -- 172.21.15.38:0/4032381971 wait complete. 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.855+0000 7f79e138e700 1 Processor -- start 2024-03-26T06:27:51.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.855+0000 7f79e138e700 1 -- start start 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.855+0000 7f79e138e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc19ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.855+0000 7f79e138e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79dc10bf50 con 0x7f79dc1079f0 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.855+0000 7f79daffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc19ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.855+0000 7f79daffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc19ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33244/0 (socket says 172.21.15.38:33244) 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.856+0000 7f79daffd700 1 -- 172.21.15.38:0/2456897753 learned_addr learned my addr 172.21.15.38:0/2456897753 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.856+0000 7f79daffd700 1 -- 172.21.15.38:0/2456897753 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc19bf50 con 0x7f79dc1079f0 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.856+0000 7f79daffd700 1 --2- 172.21.15.38:0/2456897753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc19ba10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f79cc009fb0 tx=0x7f79cc011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.856+0000 7f79c3fff700 1 -- 172.21.15.38:0/2456897753 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79cc0038b0 con 0x7f79dc1079f0 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.856+0000 7f79c3fff700 1 -- 172.21.15.38:0/2456897753 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f79cc011e50 con 0x7f79dc1079f0 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.856+0000 7f79e138e700 1 -- 172.21.15.38:0/2456897753 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79dc19c1e0 con 0x7f79dc1079f0 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.856+0000 7f79c3fff700 1 -- 172.21.15.38:0/2456897753 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f79cc019550 con 0x7f79dc1079f0 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.856+0000 7f79e138e700 1 -- 172.21.15.38:0/2456897753 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79dc19eed0 con 0x7f79dc1079f0 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.857+0000 7f79c3fff700 1 -- 172.21.15.38:0/2456897753 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f79cc01a430 con 0x7f79dc1079f0 2024-03-26T06:27:51.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.857+0000 7f79e138e700 1 -- 172.21.15.38:0/2456897753 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79dc068f20 con 0x7f79dc1079f0 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.857+0000 7f79c3fff700 1 --2- 172.21.15.38:0/2456897753 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f79c4038ed0 0x7f79c403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.857+0000 7f79c3fff700 1 -- 172.21.15.38:0/2456897753 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f79cc04cfa0 con 0x7f79dc1079f0 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.857+0000 7f79da7fc700 1 --2- 172.21.15.38:0/2456897753 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f79c4038ed0 0x7f79c403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.858+0000 7f79da7fc700 1 --2- 172.21.15.38:0/2456897753 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f79c4038ed0 0x7f79c403b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f79d0006fd0 tx=0x7f79d0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:50.862+0000 7f79c3fff700 1 -- 172.21.15.38:0/2456897753 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f79cc029260 con 0x7f79dc1079f0 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.066+0000 7f79e138e700 1 -- 172.21.15.38:0/2456897753 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f79dc19f0a0 con 0x7f79dc1079f0 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.067+0000 7f79c3fff700 1 -- 172.21.15.38:0/2456897753 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7f79dc19f0a0 con 0x7f79dc1079f0 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.069+0000 7f79e138e700 1 -- 172.21.15.38:0/2456897753 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f79c4038ed0 msgr2=0x7f79c403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.069+0000 7f79e138e700 1 --2- 172.21.15.38:0/2456897753 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f79c4038ed0 0x7f79c403b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f79d0006fd0 tx=0x7f79d0006e40 comp rx=0 tx=0).stop 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.069+0000 7f79e138e700 1 -- 172.21.15.38:0/2456897753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 msgr2=0x7f79dc19ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.069+0000 7f79e138e700 1 --2- 172.21.15.38:0/2456897753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc19ba10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f79cc009fb0 tx=0x7f79cc011bf0 comp rx=0 tx=0).stop 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.069+0000 7f79e138e700 1 -- 172.21.15.38:0/2456897753 shutdown_connections 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.069+0000 7f79e138e700 1 --2- 172.21.15.38:0/2456897753 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f79c4038ed0 0x7f79c403b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:51.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.069+0000 7f79e138e700 1 --2- 172.21.15.38:0/2456897753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79dc1079f0 0x7f79dc19ba10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.069+0000 7f79e138e700 1 -- 172.21.15.38:0/2456897753 >> 172.21.15.38:0/2456897753 conn(0x7f79dc102fd0 msgr2=0x7f79dc103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.069+0000 7f79e138e700 1 -- 172.21.15.38:0/2456897753 shutdown_connections 2024-03-26T06:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.069+0000 7f79e138e700 1 -- 172.21.15.38:0/2456897753 wait complete. 2024-03-26T06:27:51.606 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling cephadm module... 2024-03-26T06:27:51.995 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:51 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2456897753' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T06:27:53.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:52 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2905967041' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T06:27:53.083 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:52 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: ignoring --setuser ceph since I am not root 2024-03-26T06:27:53.083 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:52 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: ignoring --setgroup ceph since I am not root 2024-03-26T06:27:53.391 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.993+0000 7f47a6c3d700 1 Processor -- start 2024-03-26T06:27:53.391 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.993+0000 7f47a6c3d700 1 -- start start 2024-03-26T06:27:53.391 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.994+0000 7f47a6c3d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a0107dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:53.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.994+0000 7f47a6c3d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a01083a0 con 0x7f47a01079b0 2024-03-26T06:27:53.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.994+0000 7f47a49d9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a0107dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:53.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.994+0000 7f47a49d9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a0107dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33254/0 (socket says 172.21.15.38:33254) 2024-03-26T06:27:53.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.994+0000 7f47a49d9700 1 -- 172.21.15.38:0/2699854442 learned_addr learned my addr 172.21.15.38:0/2699854442 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:53.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.994+0000 7f47a49d9700 1 -- 172.21.15.38:0/2699854442 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a0108b60 con 0x7f47a01079b0 2024-03-26T06:27:53.392 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.994+0000 7f47a49d9700 1 --2- 172.21.15.38:0/2699854442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a0107dd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f47940098a0 tx=0x7f4794009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51426b1ffa05b133 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:53.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.994+0000 7f479f7fe700 1 -- 172.21.15.38:0/2699854442 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4794004160 con 0x7f47a01079b0 2024-03-26T06:27:53.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.994+0000 7f479f7fe700 1 -- 172.21.15.38:0/2699854442 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4794004300 con 0x7f47a01079b0 2024-03-26T06:27:53.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.994+0000 7f479f7fe700 1 -- 172.21.15.38:0/2699854442 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4794011b50 con 0x7f47a01079b0 2024-03-26T06:27:53.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.995+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2699854442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 msgr2=0x7f47a0107dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:53.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.995+0000 7f47a6c3d700 1 --2- 172.21.15.38:0/2699854442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a0107dd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f47940098a0 tx=0x7f4794009bb0 comp rx=0 tx=0).stop 2024-03-26T06:27:53.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.995+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2699854442 shutdown_connections 2024-03-26T06:27:53.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.995+0000 7f47a6c3d700 1 --2- 172.21.15.38:0/2699854442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a0107dd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:53.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.995+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2699854442 >> 172.21.15.38:0/2699854442 conn(0x7f47a0102f30 msgr2=0x7f47a0105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:53.393 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.995+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2699854442 shutdown_connections 2024-03-26T06:27:53.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.995+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2699854442 wait complete. 2024-03-26T06:27:53.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.996+0000 7f47a6c3d700 1 Processor -- start 2024-03-26T06:27:53.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.996+0000 7f47a6c3d700 1 -- start start 2024-03-26T06:27:53.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.996+0000 7f47a6c3d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a007daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:53.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.996+0000 7f47a6c3d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a010bf10 con 0x7f47a01079b0 2024-03-26T06:27:53.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.996+0000 7f47a49d9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a007daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:53.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.996+0000 7f47a49d9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a007daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33256/0 (socket says 172.21.15.38:33256) 2024-03-26T06:27:53.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.996+0000 7f47a49d9700 1 -- 172.21.15.38:0/2905967041 learned_addr learned my addr 172.21.15.38:0/2905967041 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:53.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.996+0000 7f47a49d9700 1 -- 172.21.15.38:0/2905967041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a007e030 con 0x7f47a01079b0 2024-03-26T06:27:53.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.996+0000 7f47a49d9700 1 --2- 172.21.15.38:0/2905967041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a007daf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4794011670 tx=0x7f4794011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:53.394 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.997+0000 7f479dffb700 1 -- 172.21.15.38:0/2905967041 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f47940039c0 con 0x7f47a01079b0 2024-03-26T06:27:53.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.997+0000 7f479dffb700 1 -- 172.21.15.38:0/2905967041 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4794011e50 con 0x7f47a01079b0 2024-03-26T06:27:53.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.997+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2905967041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a007c220 con 0x7f47a01079b0 2024-03-26T06:27:53.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.997+0000 7f479dffb700 1 -- 172.21.15.38:0/2905967041 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f479401a5c0 con 0x7f47a01079b0 2024-03-26T06:27:53.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.997+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2905967041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a007c760 con 0x7f47a01079b0 2024-03-26T06:27:53.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.998+0000 7f479dffb700 1 -- 172.21.15.38:0/2905967041 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f479401a870 con 0x7f47a01079b0 2024-03-26T06:27:53.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.998+0000 7f479dffb700 1 --2- 172.21.15.38:0/2905967041 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f4790038ed0 0x7f479003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:53.395 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.998+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2905967041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a0063410 con 0x7f47a01079b0 2024-03-26T06:27:53.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.998+0000 7f479dffb700 1 -- 172.21.15.38:0/2905967041 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f479404d0d0 con 0x7f47a01079b0 2024-03-26T06:27:53.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.998+0000 7f479ffff700 1 --2- 172.21.15.38:0/2905967041 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f4790038ed0 0x7f479003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:53.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:51.998+0000 7f479ffff700 1 --2- 172.21.15.38:0/2905967041 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f4790038ed0 0x7f479003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f478c006fd0 tx=0x7f478c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:53.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.003+0000 7f479dffb700 1 -- 172.21.15.38:0/2905967041 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4794019c70 con 0x7f47a01079b0 2024-03-26T06:27:53.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.269+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2905967041 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f47a007d070 con 0x7f47a01079b0 2024-03-26T06:27:53.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.791+0000 7f479dffb700 1 -- 172.21.15.38:0/2905967041 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f4794020240 con 0x7f47a01079b0 2024-03-26T06:27:53.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.793+0000 7f479dffb700 1 -- 172.21.15.38:0/2905967041 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f479402d910 con 0x7f47a01079b0 2024-03-26T06:27:53.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.795+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2905967041 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f4790038ed0 msgr2=0x7f479003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:53.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.795+0000 7f47a6c3d700 1 --2- 172.21.15.38:0/2905967041 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f4790038ed0 0x7f479003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f478c006fd0 tx=0x7f478c006e40 comp rx=0 tx=0).stop 2024-03-26T06:27:53.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.795+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2905967041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 msgr2=0x7f47a007daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:53.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.795+0000 7f47a6c3d700 1 --2- 172.21.15.38:0/2905967041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a007daf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4794011670 tx=0x7f4794011b60 comp rx=0 tx=0).stop 2024-03-26T06:27:53.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.795+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2905967041 shutdown_connections 2024-03-26T06:27:53.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.795+0000 7f47a6c3d700 1 --2- 172.21.15.38:0/2905967041 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f4790038ed0 0x7f479003b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:53.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.795+0000 7f47a6c3d700 1 --2- 172.21.15.38:0/2905967041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f47a01079b0 0x7f47a007daf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:53.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.795+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2905967041 >> 172.21.15.38:0/2905967041 conn(0x7f47a0102f30 msgr2=0x7f47a0103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:53.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.795+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2905967041 shutdown_connections 2024-03-26T06:27:53.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:52.795+0000 7f47a6c3d700 1 -- 172.21.15.38:0/2905967041 wait complete. 2024-03-26T06:27:53.471 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:53 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:53.368+0000 7f2740813200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T06:27:53.471 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:53 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:53.469+0000 7f2740813200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T06:27:54.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:53 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2905967041' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T06:27:54.044 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:53 smithi038 ceph-mon[32265]: mgrmap e5: x(active, since 6s) 2024-03-26T06:27:54.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:54 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:54.144+0000 7f2740813200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T06:27:54.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:54 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:54.242+0000 7f2740813200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T06:27:54.786 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:54 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:54.390+0000 7f2740813200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T06:27:54.787 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:54 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:54.491+0000 7f2740813200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T06:27:54.787 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:54 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:54.580+0000 7f2740813200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T06:27:54.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T06:27:54.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T06:27:54.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.837+0000 7f71cb3fa700 1 Processor -- start 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.837+0000 7f71cb3fa700 1 -- start start 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.837+0000 7f71cb3fa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c407c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.837+0000 7f71cb3fa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c407c830 con 0x7f71c407de00 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.838+0000 7f71c9196700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c407c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.838+0000 7f71c9196700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c407c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33294/0 (socket says 172.21.15.38:33294) 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.838+0000 7f71c9196700 1 -- 172.21.15.38:0/1277480922 learned_addr learned my addr 172.21.15.38:0/1277480922 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.838+0000 7f71c9196700 1 -- 172.21.15.38:0/1277480922 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c407c9b0 con 0x7f71c407de00 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.838+0000 7f71c9196700 1 --2- 172.21.15.38:0/1277480922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c407c260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f71b40098a0 tx=0x7f71b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=579b22f935bf2915 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:54.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.838+0000 7f71bbfff700 1 -- 172.21.15.38:0/1277480922 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71b4004160 con 0x7f71c407de00 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.838+0000 7f71bbfff700 1 -- 172.21.15.38:0/1277480922 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71b4004300 con 0x7f71c407de00 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.838+0000 7f71bbfff700 1 -- 172.21.15.38:0/1277480922 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71b4011b50 con 0x7f71c407de00 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.839+0000 7f71cb3fa700 1 -- 172.21.15.38:0/1277480922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 msgr2=0x7f71c407c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.839+0000 7f71cb3fa700 1 --2- 172.21.15.38:0/1277480922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c407c260 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f71b40098a0 tx=0x7f71b4009bb0 comp rx=0 tx=0).stop 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.839+0000 7f71cb3fa700 1 -- 172.21.15.38:0/1277480922 shutdown_connections 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.839+0000 7f71cb3fa700 1 --2- 172.21.15.38:0/1277480922 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c407c260 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.839+0000 7f71cb3fa700 1 -- 172.21.15.38:0/1277480922 >> 172.21.15.38:0/1277480922 conn(0x7f71c4102f70 msgr2=0x7f71c4105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.839+0000 7f71cb3fa700 1 -- 172.21.15.38:0/1277480922 shutdown_connections 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.839+0000 7f71cb3fa700 1 -- 172.21.15.38:0/1277480922 wait complete. 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.840+0000 7f71cb3fa700 1 Processor -- start 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.840+0000 7f71cb3fa700 1 -- start start 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.840+0000 7f71cb3fa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c419dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:54.797 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.840+0000 7f71cb3fa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c410dfc0 con 0x7f71c407de00 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.840+0000 7f71c9196700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c419dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.840+0000 7f71c9196700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c419dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33298/0 (socket says 172.21.15.38:33298) 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.840+0000 7f71c9196700 1 -- 172.21.15.38:0/2859886180 learned_addr learned my addr 172.21.15.38:0/2859886180 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.840+0000 7f71c9196700 1 -- 172.21.15.38:0/2859886180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c419e0e0 con 0x7f71c407de00 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.840+0000 7f71c9196700 1 --2- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c419dba0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f71b4009fb0 tx=0x7f71b4011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.841+0000 7f71ba7fc700 1 -- 172.21.15.38:0/2859886180 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71b40038b0 con 0x7f71c407de00 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.841+0000 7f71ba7fc700 1 -- 172.21.15.38:0/2859886180 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f71b4011e50 con 0x7f71c407de00 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.841+0000 7f71ba7fc700 1 -- 172.21.15.38:0/2859886180 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f71b4019420 con 0x7f71c407de00 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.841+0000 7f71cb3fa700 1 -- 172.21.15.38:0/2859886180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c419e370 con 0x7f71c407de00 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.841+0000 7f71cb3fa700 1 -- 172.21.15.38:0/2859886180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c41a1060 con 0x7f71c407de00 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.842+0000 7f71ba7fc700 1 -- 172.21.15.38:0/2859886180 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f71b4019720 con 0x7f71c407de00 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.842+0000 7f71ba7fc700 1 --2- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f71b0038ed0 0x7f71b003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:27:54.798 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.842+0000 7f71cb3fa700 1 -- 172.21.15.38:0/2859886180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c4068f20 con 0x7f71c407de00 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.842+0000 7f71ba7fc700 1 -- 172.21.15.38:0/2859886180 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f71b404d050 con 0x7f71c407de00 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.842+0000 7f71c8995700 1 -- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f71b0038ed0 msgr2=0x7f71b003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/1985135432 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.842+0000 7f71c8995700 1 --2- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f71b0038ed0 0x7f71b003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:53.847+0000 7f71ba7fc700 1 -- 172.21.15.38:0/2859886180 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f71b4003a50 con 0x7f71c407de00 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.043+0000 7f71c8995700 1 -- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f71b0038ed0 msgr2=0x7f71b003b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/1985135432 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.043+0000 7f71c8995700 1 --2- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f71b0038ed0 0x7f71b003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.109+0000 7f71cb3fa700 1 -- 172.21.15.38:0/2859886180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f71c41a1230 con 0x7f71c407de00 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.110+0000 7f71ba7fc700 1 -- 172.21.15.38:0/2859886180 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f71c41a1230 con 0x7f71c407de00 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.112+0000 7f71cb3fa700 1 -- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f71b0038ed0 msgr2=0x7f71b003b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.112+0000 7f71cb3fa700 1 --2- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f71b0038ed0 0x7f71b003b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.112+0000 7f71cb3fa700 1 -- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 msgr2=0x7f71c419dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:27:54.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.112+0000 7f71cb3fa700 1 --2- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c419dba0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f71b4009fb0 tx=0x7f71b4011bf0 comp rx=0 tx=0).stop 2024-03-26T06:27:54.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.112+0000 7f71cb3fa700 1 -- 172.21.15.38:0/2859886180 shutdown_connections 2024-03-26T06:27:54.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.112+0000 7f71cb3fa700 1 --2- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7f71b0038ed0 0x7f71b003b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:54.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.112+0000 7f71cb3fa700 1 --2- 172.21.15.38:0/2859886180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71c407de00 0x7f71c419dba0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:27:54.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.112+0000 7f71cb3fa700 1 -- 172.21.15.38:0/2859886180 >> 172.21.15.38:0/2859886180 conn(0x7f71c4102f70 msgr2=0x7f71c4103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:27:54.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.112+0000 7f71cb3fa700 1 -- 172.21.15.38:0/2859886180 shutdown_connections 2024-03-26T06:27:54.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:54.112+0000 7f71cb3fa700 1 -- 172.21.15.38:0/2859886180 wait complete. 2024-03-26T06:27:54.803 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-03-26T06:27:54.803 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 5... 2024-03-26T06:27:55.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:54 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2859886180' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T06:27:55.072 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:54 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:54.859+0000 7f2740813200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T06:27:55.072 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:54 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:54.974+0000 7f2740813200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T06:27:55.072 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:55 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:55.069+0000 7f2740813200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T06:27:55.332 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:55 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:55.168+0000 7f2740813200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T06:27:55.832 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:55 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:55.351+0000 7f2740813200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T06:27:56.633 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:56 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:56.337+0000 7f2740813200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T06:27:56.634 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:56 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:56.470+0000 7f2740813200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T06:27:56.906 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:56 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:56.812+0000 7f2740813200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T06:27:56.906 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:56 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:56.904+0000 7f2740813200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T06:27:57.292 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:57 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:57.103+0000 7f2740813200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:27:57.292 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:57 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:57.290+0000 7f2740813200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T06:27:57.832 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:57 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:57.579+0000 7f2740813200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:27:58.235 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:57 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:57.840+0000 7f2740813200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T06:27:58.235 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:58 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:58.080+0000 7f2740813200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T06:27:59.332 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:58 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:58.902+0000 7f2740813200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T06:27:59.806 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:59.484+0000 7f2740813200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T06:27:59.806 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:59.584+0000 7f2740813200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T06:27:59.806 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:27:59.804+0000 7f2740813200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T06:28:00.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-mon[32265]: Active manager daemon x restarted 2024-03-26T06:28:00.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-mon[32265]: Activating manager daemon x 2024-03-26T06:28:00.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-mon[32265]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T06:28:00.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-mon[32265]: mgrmap e6: x(active, starting, since 0.0123351s) 2024-03-26T06:28:00.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:28:00.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T06:28:00.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T06:28:00.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T06:28:00.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T06:28:00.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-mon[32265]: Manager daemon x is now available 2024-03-26T06:28:00.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:27:59 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:01.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: Found migration_current of "None". Setting to last migration. 2024-03-26T06:28:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T06:28:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T06:28:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T06:28:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: mgrmap e7: x(active, since 1.01132s) 2024-03-26T06:28:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:00 smithi038 ceph-mon[32265]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T06:28:01.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T06:28:01.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T06:28:01.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T06:28:01.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T06:28:01.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.224+0000 7fb637b33700 1 Processor -- start 2024-03-26T06:28:01.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.224+0000 7fb637b33700 1 -- start start 2024-03-26T06:28:01.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.225+0000 7fb637b33700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb63010a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:01.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.225+0000 7fb637b33700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb63007bdd0 con 0x7fb630108460 2024-03-26T06:28:01.396 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.225+0000 7fb6358cf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb63010a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.225+0000 7fb6358cf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb63010a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42876/0 (socket says 172.21.15.38:42876) 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.225+0000 7fb6358cf700 1 -- 172.21.15.38:0/3154802336 learned_addr learned my addr 172.21.15.38:0/3154802336 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.225+0000 7fb6358cf700 1 -- 172.21.15.38:0/3154802336 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63010ad90 con 0x7fb630108460 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.225+0000 7fb6358cf700 1 --2- 172.21.15.38:0/3154802336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb63010a850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0098a0 tx=0x7fb62c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b8b9448434f8cd server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.225+0000 7fb6348cd700 1 -- 172.21.15.38:0/3154802336 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb62c004160 con 0x7fb630108460 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.225+0000 7fb6348cd700 1 -- 172.21.15.38:0/3154802336 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb62c004300 con 0x7fb630108460 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.225+0000 7fb6348cd700 1 -- 172.21.15.38:0/3154802336 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb62c011b50 con 0x7fb630108460 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.226+0000 7fb637b33700 1 -- 172.21.15.38:0/3154802336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 msgr2=0x7fb63010a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.226+0000 7fb637b33700 1 --2- 172.21.15.38:0/3154802336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb63010a850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0098a0 tx=0x7fb62c009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.226+0000 7fb637b33700 1 -- 172.21.15.38:0/3154802336 shutdown_connections 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.226+0000 7fb637b33700 1 --2- 172.21.15.38:0/3154802336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb63010a850 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.226+0000 7fb637b33700 1 -- 172.21.15.38:0/3154802336 >> 172.21.15.38:0/3154802336 conn(0x7fb630101e50 msgr2=0x7fb630104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:01.397 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.226+0000 7fb637b33700 1 -- 172.21.15.38:0/3154802336 shutdown_connections 2024-03-26T06:28:01.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.226+0000 7fb637b33700 1 -- 172.21.15.38:0/3154802336 wait complete. 2024-03-26T06:28:01.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.227+0000 7fb637b33700 1 Processor -- start 2024-03-26T06:28:01.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.227+0000 7fb637b33700 1 -- start start 2024-03-26T06:28:01.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.227+0000 7fb637b33700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb630199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:01.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.227+0000 7fb637b33700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb630106e00 con 0x7fb630108460 2024-03-26T06:28:01.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.227+0000 7fb6358cf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb630199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:01.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.227+0000 7fb6358cf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb630199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42892/0 (socket says 172.21.15.38:42892) 2024-03-26T06:28:01.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.227+0000 7fb6358cf700 1 -- 172.21.15.38:0/474258110 learned_addr learned my addr 172.21.15.38:0/474258110 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:01.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.227+0000 7fb6358cf700 1 -- 172.21.15.38:0/474258110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb630199c60 con 0x7fb630108460 2024-03-26T06:28:01.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.228+0000 7fb6358cf700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb630199720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009fb0 tx=0x7fb62c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:01.398 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.228+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb62c0038b0 con 0x7fb630108460 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.228+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb62c011e50 con 0x7fb630108460 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.228+0000 7fb637b33700 1 -- 172.21.15.38:0/474258110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb630199ef0 con 0x7fb630108460 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.228+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb62c019420 con 0x7fb630108460 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.228+0000 7fb637b33700 1 -- 172.21.15.38:0/474258110 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb63019cbe0 con 0x7fb630108460 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.229+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb62c019720 con 0x7fb630108460 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.229+0000 7fb626ffd700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 0x7fb61c03f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.229+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 --> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb61c03feb0 con 0x7fb61c03d2e0 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.229+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb62c04d1e0 con 0x7fb630108460 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.229+0000 7fb6350ce700 1 -- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 msgr2=0x7fb61c03f7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/1985135432 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.229+0000 7fb6350ce700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 0x7fb61c03f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.430+0000 7fb6350ce700 1 -- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 msgr2=0x7fb61c03f7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/1985135432 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.430+0000 7fb6350ce700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 0x7fb61c03f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T06:28:01.399 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.831+0000 7fb6350ce700 1 -- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 msgr2=0x7fb61c03f7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/1985135432 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:55.831+0000 7fb6350ce700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 0x7fb61c03f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:56.632+0000 7fb6350ce700 1 -- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 msgr2=0x7fb61c03f7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/1985135432 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:56.632+0000 7fb6350ce700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 0x7fb61c03f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:58.233+0000 7fb6350ce700 1 -- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 msgr2=0x7fb61c03f7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/1985135432 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:58.233+0000 7fb6350ce700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 0x7fb61c03f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:59.828+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7fb62c019a60 con 0x7fb630108460 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:59.828+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 msgr2=0x7fb61c03f7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:27:59.828+0000 7fb626ffd700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 0x7fb61c03f7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.826+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7fb62c04ed30 con 0x7fb630108460 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.826+0000 7fb626ffd700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fb61c040bd0 0x7fb61c042fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.826+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 --> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb62c001d30 con 0x7fb61c040bd0 2024-03-26T06:28:01.400 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.827+0000 7fb6350ce700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fb61c040bd0 0x7fb61c042fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.827+0000 7fb6350ce700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fb61c040bd0 0x7fb61c042fc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb620003a10 tx=0x7fb620009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.828+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 <== mgr.14120 v2:172.21.15.38:6800/4246298230 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fb62c001d30 con 0x7fb61c040bd0 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.834+0000 7fb637b33700 1 -- 172.21.15.38:0/474258110 --> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb630068f20 con 0x7fb61c040bd0 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.834+0000 7fb626ffd700 1 -- 172.21.15.38:0/474258110 <== mgr.14120 v2:172.21.15.38:6800/4246298230 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb630068f20 con 0x7fb61c040bd0 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.835+0000 7fb637b33700 1 -- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fb61c040bd0 msgr2=0x7fb61c042fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.835+0000 7fb637b33700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fb61c040bd0 0x7fb61c042fc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb620003a10 tx=0x7fb620009290 comp rx=0 tx=0).stop 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.835+0000 7fb637b33700 1 -- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 msgr2=0x7fb630199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.835+0000 7fb637b33700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb630199720 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009fb0 tx=0x7fb62c011bf0 comp rx=0 tx=0).stop 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.835+0000 7fb637b33700 1 -- 172.21.15.38:0/474258110 shutdown_connections 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.835+0000 7fb637b33700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fb61c040bd0 0x7fb61c042fc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.835+0000 7fb637b33700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb630108460 0x7fb630199720 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.835+0000 7fb637b33700 1 --2- 172.21.15.38:0/474258110 >> [v2:172.21.15.38:6800/1985135432,v1:172.21.15.38:6801/1985135432] conn(0x7fb61c03d2e0 0x7fb61c03f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:01.401 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.835+0000 7fb637b33700 1 -- 172.21.15.38:0/474258110 >> 172.21.15.38:0/474258110 conn(0x7fb630101e50 msgr2=0x7fb6301026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:01.402 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.835+0000 7fb637b33700 1 -- 172.21.15.38:0/474258110 shutdown_connections 2024-03-26T06:28:01.402 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:00.835+0000 7fb637b33700 1 -- 172.21.15.38:0/474258110 wait complete. 2024-03-26T06:28:01.402 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 5 is available 2024-03-26T06:28:01.402 INFO:teuthology.orchestra.run.smithi038.stdout:Setting orchestrator backend to cephadm... 2024-03-26T06:28:02.038 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:01 smithi038 ceph-mon[32265]: [26/Mar/2024:06:28:01] ENGINE Bus STARTING 2024-03-26T06:28:02.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:01 smithi038 ceph-mon[32265]: [26/Mar/2024:06:28:01] ENGINE Serving on https://172.21.15.38:7150 2024-03-26T06:28:02.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:01 smithi038 ceph-mon[32265]: [26/Mar/2024:06:28:01] ENGINE Serving on http://172.21.15.38:8765 2024-03-26T06:28:02.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:01 smithi038 ceph-mon[32265]: [26/Mar/2024:06:28:01] ENGINE Bus STARTED 2024-03-26T06:28:02.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:01 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:02.549 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.806+0000 7fbec53e6700 1 Processor -- start 2024-03-26T06:28:02.549 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.806+0000 7fbec53e6700 1 -- start start 2024-03-26T06:28:02.549 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.806+0000 7fbec53e6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0108290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:02.549 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.806+0000 7fbec53e6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec0108860 con 0x7fbec0107e70 2024-03-26T06:28:02.549 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.807+0000 7fbebeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0108290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:02.549 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.807+0000 7fbebeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0108290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42974/0 (socket says 172.21.15.38:42974) 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.807+0000 7fbebeffd700 1 -- 172.21.15.38:0/4119766588 learned_addr learned my addr 172.21.15.38:0/4119766588 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.807+0000 7fbebeffd700 1 -- 172.21.15.38:0/4119766588 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec0106b20 con 0x7fbec0107e70 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.807+0000 7fbebeffd700 1 --2- 172.21.15.38:0/4119766588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0108290 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009e80 tx=0x7fbeb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eafea371bb42167d server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.807+0000 7fbebdffb700 1 -- 172.21.15.38:0/4119766588 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbeb8017070 con 0x7fbec0107e70 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.807+0000 7fbebdffb700 1 -- 172.21.15.38:0/4119766588 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbeb800b8c0 con 0x7fbec0107e70 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.807+0000 7fbebdffb700 1 -- 172.21.15.38:0/4119766588 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbeb80045c0 con 0x7fbec0107e70 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.808+0000 7fbec53e6700 1 -- 172.21.15.38:0/4119766588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 msgr2=0x7fbec0108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.808+0000 7fbec53e6700 1 --2- 172.21.15.38:0/4119766588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0108290 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009e80 tx=0x7fbeb800b330 comp rx=0 tx=0).stop 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.808+0000 7fbec53e6700 1 -- 172.21.15.38:0/4119766588 shutdown_connections 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.808+0000 7fbec53e6700 1 --2- 172.21.15.38:0/4119766588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0108290 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.808+0000 7fbec53e6700 1 -- 172.21.15.38:0/4119766588 >> 172.21.15.38:0/4119766588 conn(0x7fbec01015f0 msgr2=0x7fbec0103a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.808+0000 7fbec53e6700 1 -- 172.21.15.38:0/4119766588 shutdown_connections 2024-03-26T06:28:02.550 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.808+0000 7fbec53e6700 1 -- 172.21.15.38:0/4119766588 wait complete. 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.808+0000 7fbec53e6700 1 Processor -- start 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbec53e6700 1 -- start start 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbec53e6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0106490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbec53e6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec010aee0 con 0x7fbec0107e70 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbebeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbebeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42988/0 (socket says 172.21.15.38:42988) 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbebeffd700 1 -- 172.21.15.38:0/57571552 learned_addr learned my addr 172.21.15.38:0/57571552 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbebeffd700 1 -- 172.21.15.38:0/57571552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbec019e670 con 0x7fbec0107e70 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbebeffd700 1 --2- 172.21.15.38:0/57571552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0106490 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009530 tx=0x7fbeb800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbeb7fff700 1 -- 172.21.15.38:0/57571552 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbeb8020020 con 0x7fbec0107e70 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbeb7fff700 1 -- 172.21.15.38:0/57571552 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbeb8003b50 con 0x7fbec0107e70 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbec53e6700 1 -- 172.21.15.38:0/57571552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec019e840 con 0x7fbec0107e70 2024-03-26T06:28:02.551 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbeb7fff700 1 -- 172.21.15.38:0/57571552 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbeb8017040 con 0x7fbec0107e70 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.809+0000 7fbec53e6700 1 -- 172.21.15.38:0/57571552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec019ec60 con 0x7fbec0107e70 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.810+0000 7fbeb7fff700 1 -- 172.21.15.38:0/57571552 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7fbeb8023020 con 0x7fbec0107e70 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.810+0000 7fbeb7fff700 1 --2- 172.21.15.38:0/57571552 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbeac038e50 0x7fbeac03b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.810+0000 7fbec53e6700 1 -- 172.21.15.38:0/57571552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec0063410 con 0x7fbec0107e70 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.810+0000 7fbeb7fff700 1 -- 172.21.15.38:0/57571552 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbeb800f050 con 0x7fbec0107e70 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.811+0000 7fbebe7fc700 1 --2- 172.21.15.38:0/57571552 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbeac038e50 0x7fbeac03b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.811+0000 7fbebe7fc700 1 --2- 172.21.15.38:0/57571552 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbeac038e50 0x7fbeac03b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbea8006fd0 tx=0x7fbea8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.816+0000 7fbeb7fff700 1 -- 172.21.15.38:0/57571552 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbeb802a020 con 0x7fbec0107e70 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:01.874+0000 7fbeb7fff700 1 -- 172.21.15.38:0/57571552 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fbeb804b7d0 con 0x7fbec0107e70 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.037+0000 7fbec53e6700 1 -- 172.21.15.38:0/57571552 --> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fbec0102850 con 0x7fbeac038e50 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.044+0000 7fbeb7fff700 1 -- 172.21.15.38:0/57571552 <== mgr.14120 v2:172.21.15.38:6800/4246298230 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbec0102850 con 0x7fbeac038e50 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.047+0000 7fbec53e6700 1 -- 172.21.15.38:0/57571552 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbeac038e50 msgr2=0x7fbeac03b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.047+0000 7fbec53e6700 1 --2- 172.21.15.38:0/57571552 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbeac038e50 0x7fbeac03b310 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbea8006fd0 tx=0x7fbea8006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:02.552 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.047+0000 7fbec53e6700 1 -- 172.21.15.38:0/57571552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 msgr2=0x7fbec0106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:02.553 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.047+0000 7fbec53e6700 1 --2- 172.21.15.38:0/57571552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0106490 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009530 tx=0x7fbeb800bd50 comp rx=0 tx=0).stop 2024-03-26T06:28:02.553 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.047+0000 7fbec53e6700 1 -- 172.21.15.38:0/57571552 shutdown_connections 2024-03-26T06:28:02.553 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.047+0000 7fbec53e6700 1 --2- 172.21.15.38:0/57571552 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbeac038e50 0x7fbeac03b310 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:02.553 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.047+0000 7fbec53e6700 1 --2- 172.21.15.38:0/57571552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbec0107e70 0x7fbec0106490 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:02.553 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.047+0000 7fbec53e6700 1 -- 172.21.15.38:0/57571552 >> 172.21.15.38:0/57571552 conn(0x7fbec01015f0 msgr2=0x7fbec0102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:02.553 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.047+0000 7fbec53e6700 1 -- 172.21.15.38:0/57571552 shutdown_connections 2024-03-26T06:28:02.553 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.047+0000 7fbec53e6700 1 -- 172.21.15.38:0/57571552 wait complete. 2024-03-26T06:28:02.920 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:02 smithi038 ceph-mon[32265]: mgrmap e8: x(active, since 2s) 2024-03-26T06:28:02.920 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:02 smithi038 ceph-mon[32265]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:02.920 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:02 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:02.920 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:02 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:03.714 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.948+0000 7fbbf4293700 1 Processor -- start 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.948+0000 7fbbf4293700 1 -- start start 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.948+0000 7fbbf4293700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec106690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.948+0000 7fbbf4293700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec07bdb0 con 0x7fbbec106270 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.948+0000 7fbbf202f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec106690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.948+0000 7fbbf202f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec106690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43000/0 (socket says 172.21.15.38:43000) 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.948+0000 7fbbf202f700 1 -- 172.21.15.38:0/1265066631 learned_addr learned my addr 172.21.15.38:0/1265066631 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.949+0000 7fbbf202f700 1 -- 172.21.15.38:0/1265066631 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec106bd0 con 0x7fbbec106270 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.949+0000 7fbbf202f700 1 --2- 172.21.15.38:0/1265066631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec106690 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009e80 tx=0x7fbbe400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794f7cdabddfecfc server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.949+0000 7fbbf102d700 1 -- 172.21.15.38:0/1265066631 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbbe4017070 con 0x7fbbec106270 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.949+0000 7fbbf102d700 1 -- 172.21.15.38:0/1265066631 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbe400b8c0 con 0x7fbbec106270 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.949+0000 7fbbf102d700 1 -- 172.21.15.38:0/1265066631 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbbe40045c0 con 0x7fbbec106270 2024-03-26T06:28:03.715 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.950+0000 7fbbf4293700 1 -- 172.21.15.38:0/1265066631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 msgr2=0x7fbbec106690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.950+0000 7fbbf4293700 1 --2- 172.21.15.38:0/1265066631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec106690 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009e80 tx=0x7fbbe400b330 comp rx=0 tx=0).stop 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.950+0000 7fbbf4293700 1 -- 172.21.15.38:0/1265066631 shutdown_connections 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.950+0000 7fbbf4293700 1 --2- 172.21.15.38:0/1265066631 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec106690 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.950+0000 7fbbf4293700 1 -- 172.21.15.38:0/1265066631 >> 172.21.15.38:0/1265066631 conn(0x7fbbec101e10 msgr2=0x7fbbec104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.950+0000 7fbbf4293700 1 -- 172.21.15.38:0/1265066631 shutdown_connections 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.950+0000 7fbbf4293700 1 -- 172.21.15.38:0/1265066631 wait complete. 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.950+0000 7fbbf4293700 1 Processor -- start 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.950+0000 7fbbf4293700 1 -- start start 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.950+0000 7fbbf4293700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec19db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.950+0000 7fbbf4293700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbec107c60 con 0x7fbbec106270 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.951+0000 7fbbf202f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec19db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:03.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.951+0000 7fbbf202f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec19db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43012/0 (socket says 172.21.15.38:43012) 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.951+0000 7fbbf202f700 1 -- 172.21.15.38:0/1694724617 learned_addr learned my addr 172.21.15.38:0/1694724617 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.951+0000 7fbbf202f700 1 -- 172.21.15.38:0/1694724617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbec19e050 con 0x7fbbec106270 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.951+0000 7fbbf202f700 1 --2- 172.21.15.38:0/1694724617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec19db10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009530 tx=0x7fbbe400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.951+0000 7fbbeb7fe700 1 -- 172.21.15.38:0/1694724617 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbbe4020020 con 0x7fbbec106270 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.951+0000 7fbbf4293700 1 -- 172.21.15.38:0/1694724617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbec19e2e0 con 0x7fbbec106270 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.951+0000 7fbbeb7fe700 1 -- 172.21.15.38:0/1694724617 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbbe4003b50 con 0x7fbbec106270 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.951+0000 7fbbeb7fe700 1 -- 172.21.15.38:0/1694724617 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbbe4017040 con 0x7fbbec106270 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.951+0000 7fbbf4293700 1 -- 172.21.15.38:0/1694724617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbec1a0fd0 con 0x7fbbec106270 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.952+0000 7fbbeb7fe700 1 -- 172.21.15.38:0/1694724617 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fbbe4023020 con 0x7fbbec106270 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.952+0000 7fbbf4293700 1 -- 172.21.15.38:0/1694724617 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbec068f20 con 0x7fbbec106270 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.952+0000 7fbbeb7fe700 1 --2- 172.21.15.38:0/1694724617 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbbd4038f20 0x7fbbd403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:03.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.952+0000 7fbbeb7fe700 1 -- 172.21.15.38:0/1694724617 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbbe400f050 con 0x7fbbec106270 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.953+0000 7fbbf182e700 1 --2- 172.21.15.38:0/1694724617 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbbd4038f20 0x7fbbd403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.953+0000 7fbbf182e700 1 --2- 172.21.15.38:0/1694724617 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbbd4038f20 0x7fbbd403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0006fd0 tx=0x7fbbe0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:02.958+0000 7fbbeb7fe700 1 -- 172.21.15.38:0/1694724617 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbbe4053080 con 0x7fbbec106270 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.164+0000 7fbbf4293700 1 -- 172.21.15.38:0/1694724617 --> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbec02db10 con 0x7fbbd4038f20 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.164+0000 7fbbeb7fe700 1 -- 172.21.15.38:0/1694724617 <== mgr.14120 v2:172.21.15.38:6800/4246298230 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fbbec02db10 con 0x7fbbd4038f20 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.167+0000 7fbbf4293700 1 -- 172.21.15.38:0/1694724617 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbbd4038f20 msgr2=0x7fbbd403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.167+0000 7fbbf4293700 1 --2- 172.21.15.38:0/1694724617 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbbd4038f20 0x7fbbd403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0006fd0 tx=0x7fbbe0006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.167+0000 7fbbf4293700 1 -- 172.21.15.38:0/1694724617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 msgr2=0x7fbbec19db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.167+0000 7fbbf4293700 1 --2- 172.21.15.38:0/1694724617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec19db10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4009530 tx=0x7fbbe400bd50 comp rx=0 tx=0).stop 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.167+0000 7fbbf4293700 1 -- 172.21.15.38:0/1694724617 shutdown_connections 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.167+0000 7fbbf4293700 1 --2- 172.21.15.38:0/1694724617 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbbd4038f20 0x7fbbd403b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:03.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.167+0000 7fbbf4293700 1 --2- 172.21.15.38:0/1694724617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbbec106270 0x7fbbec19db10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:03.719 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.167+0000 7fbbf4293700 1 -- 172.21.15.38:0/1694724617 >> 172.21.15.38:0/1694724617 conn(0x7fbbec101e10 msgr2=0x7fbbec194730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:03.719 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.167+0000 7fbbf4293700 1 -- 172.21.15.38:0/1694724617 shutdown_connections 2024-03-26T06:28:03.719 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:03.167+0000 7fbbf4293700 1 -- 172.21.15.38:0/1694724617 wait complete. 2024-03-26T06:28:03.719 INFO:teuthology.orchestra.run.smithi038.stdout:Generating ssh key... 2024-03-26T06:28:04.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:03 smithi038 ceph-mon[32265]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:04.860 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: Generating public/private rsa key pair. 2024-03-26T06:28:04.860 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: Your identification has been saved in /tmp/tmp5pesjww9/key. 2024-03-26T06:28:04.860 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: Your public key has been saved in /tmp/tmp5pesjww9/key.pub. 2024-03-26T06:28:04.860 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: The key fingerprint is: 2024-03-26T06:28:04.860 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: SHA256:I7fWvUgimOtJuHTu7GVGlBpwoYTD64MHheRKJcJUeBY ceph-c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:28:04.860 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: The key's randomart image is: 2024-03-26T06:28:04.860 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: +---[RSA 3072]----+ 2024-03-26T06:28:04.860 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: |**=E+. | 2024-03-26T06:28:04.860 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: |*=+* . | 2024-03-26T06:28:04.861 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: | == . o | 2024-03-26T06:28:04.861 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: |+. + | 2024-03-26T06:28:04.861 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: |+. . o S | 2024-03-26T06:28:04.861 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: |.o.. + o + . | 2024-03-26T06:28:04.861 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: | .+ = = + o . | 2024-03-26T06:28:04.861 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: | . * * o o . . | 2024-03-26T06:28:04.861 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: | .+O . . | 2024-03-26T06:28:04.861 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: +----[SHA256]-----+ 2024-03-26T06:28:05.146 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.138+0000 7f3b04f1a700 1 Processor -- start 2024-03-26T06:28:05.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.139+0000 7f3b04f1a700 1 -- start start 2024-03-26T06:28:05.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.139+0000 7f3b04f1a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b00106ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:05.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.139+0000 7f3b04f1a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b00107270 con 0x7f3b00106880 2024-03-26T06:28:05.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.139+0000 7f3afe59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b00106ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:05.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.139+0000 7f3afe59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b00106ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43026/0 (socket says 172.21.15.38:43026) 2024-03-26T06:28:05.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.139+0000 7f3afe59c700 1 -- 172.21.15.38:0/492769508 learned_addr learned my addr 172.21.15.38:0/492769508 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:05.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.139+0000 7f3afe59c700 1 -- 172.21.15.38:0/492769508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b00107a80 con 0x7f3b00106880 2024-03-26T06:28:05.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.139+0000 7f3afe59c700 1 --2- 172.21.15.38:0/492769508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b00106ca0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3af00098a0 tx=0x7f3af0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4c4e20880977d2a server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:05.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.140+0000 7f3afd59a700 1 -- 172.21.15.38:0/492769508 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3af0004160 con 0x7f3b00106880 2024-03-26T06:28:05.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.140+0000 7f3afd59a700 1 -- 172.21.15.38:0/492769508 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3af0004300 con 0x7f3b00106880 2024-03-26T06:28:05.147 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.140+0000 7f3afd59a700 1 -- 172.21.15.38:0/492769508 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3af0011b50 con 0x7f3b00106880 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.140+0000 7f3b04f1a700 1 -- 172.21.15.38:0/492769508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 msgr2=0x7f3b00106ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.140+0000 7f3b04f1a700 1 --2- 172.21.15.38:0/492769508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b00106ca0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f3af00098a0 tx=0x7f3af0009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.140+0000 7f3b04f1a700 1 -- 172.21.15.38:0/492769508 shutdown_connections 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.140+0000 7f3b04f1a700 1 --2- 172.21.15.38:0/492769508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b00106ca0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.140+0000 7f3b04f1a700 1 -- 172.21.15.38:0/492769508 >> 172.21.15.38:0/492769508 conn(0x7f3b00101e40 msgr2=0x7f3b00104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.140+0000 7f3b04f1a700 1 -- 172.21.15.38:0/492769508 shutdown_connections 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.140+0000 7f3b04f1a700 1 -- 172.21.15.38:0/492769508 wait complete. 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.141+0000 7f3b04f1a700 1 Processor -- start 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.141+0000 7f3b04f1a700 1 -- start start 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.141+0000 7f3b04f1a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b0007b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.141+0000 7f3b04f1a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0010ae30 con 0x7f3b00106880 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.141+0000 7f3afe59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b0007b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:05.148 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.141+0000 7f3afe59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b0007b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43030/0 (socket says 172.21.15.38:43030) 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.141+0000 7f3afe59c700 1 -- 172.21.15.38:0/2896992725 learned_addr learned my addr 172.21.15.38:0/2896992725 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.142+0000 7f3afe59c700 1 -- 172.21.15.38:0/2896992725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0019e710 con 0x7f3b00106880 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.142+0000 7f3afe59c700 1 --2- 172.21.15.38:0/2896992725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b0007b9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009fb0 tx=0x7f3af0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.142+0000 7f3aef7fe700 1 -- 172.21.15.38:0/2896992725 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3af00038b0 con 0x7f3b00106880 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.142+0000 7f3aef7fe700 1 -- 172.21.15.38:0/2896992725 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3af0011e50 con 0x7f3b00106880 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.142+0000 7f3b04f1a700 1 -- 172.21.15.38:0/2896992725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0019e8e0 con 0x7f3b00106880 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.142+0000 7f3aef7fe700 1 -- 172.21.15.38:0/2896992725 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3af0019550 con 0x7f3b00106880 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.142+0000 7f3b04f1a700 1 -- 172.21.15.38:0/2896992725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0019ec80 con 0x7f3b00106880 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.143+0000 7f3aef7fe700 1 -- 172.21.15.38:0/2896992725 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f3af0019800 con 0x7f3b00106880 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.143+0000 7f3b04f1a700 1 -- 172.21.15.38:0/2896992725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b00068f20 con 0x7f3b00106880 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.143+0000 7f3aef7fe700 1 --2- 172.21.15.38:0/2896992725 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f3ae8038f20 0x7f3ae803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.143+0000 7f3aef7fe700 1 -- 172.21.15.38:0/2896992725 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f3af004d2f0 con 0x7f3b00106880 2024-03-26T06:28:05.149 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.143+0000 7f3afdd9b700 1 --2- 172.21.15.38:0/2896992725 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f3ae8038f20 0x7f3ae803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.144+0000 7f3afdd9b700 1 --2- 172.21.15.38:0/2896992725 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f3ae8038f20 0x7f3ae803b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3af4006fd0 tx=0x7f3af4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.148+0000 7f3aef7fe700 1 -- 172.21.15.38:0/2896992725 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3af0003ac0 con 0x7f3b00106880 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.351+0000 7f3b04f1a700 1 -- 172.21.15.38:0/2896992725 --> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b00103100 con 0x7f3ae8038f20 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.610+0000 7f3aef7fe700 1 -- 172.21.15.38:0/2896992725 <== mgr.14120 v2:172.21.15.38:6800/4246298230 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3b00103100 con 0x7f3ae8038f20 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.612+0000 7f3b04f1a700 1 -- 172.21.15.38:0/2896992725 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f3ae8038f20 msgr2=0x7f3ae803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.612+0000 7f3b04f1a700 1 --2- 172.21.15.38:0/2896992725 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f3ae8038f20 0x7f3ae803b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3af4006fd0 tx=0x7f3af4006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.612+0000 7f3b04f1a700 1 -- 172.21.15.38:0/2896992725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 msgr2=0x7f3b0007b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.612+0000 7f3b04f1a700 1 --2- 172.21.15.38:0/2896992725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b0007b9d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009fb0 tx=0x7f3af0011bf0 comp rx=0 tx=0).stop 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.612+0000 7f3b04f1a700 1 -- 172.21.15.38:0/2896992725 shutdown_connections 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.612+0000 7f3b04f1a700 1 --2- 172.21.15.38:0/2896992725 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f3ae8038f20 0x7f3ae803b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.612+0000 7f3b04f1a700 1 --2- 172.21.15.38:0/2896992725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3b00106880 0x7f3b0007b9d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.612+0000 7f3b04f1a700 1 -- 172.21.15.38:0/2896992725 >> 172.21.15.38:0/2896992725 conn(0x7f3b00101e40 msgr2=0x7f3b001029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:05.150 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.612+0000 7f3b04f1a700 1 -- 172.21.15.38:0/2896992725 shutdown_connections 2024-03-26T06:28:05.151 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:04.613+0000 7f3b04f1a700 1 -- 172.21.15.38:0/2896992725 wait complete. 2024-03-26T06:28:05.766 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:05 smithi038 ceph-mon[32265]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:05.766 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:05 smithi038 ceph-mon[32265]: Generating ssh key... 2024-03-26T06:28:05.766 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:05 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:05.766 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:05 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:06.330 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS8CLzDJbbKJNW6ZfsKSGWqdnZgvFnZTZB2X4o2Pc0FvC7Z80cgLODE8fkK5VVevZlSHJ1jRr6I7grFmSZgeFxTdqpOghjuvjlMgt0OaXicSIg7EVA1P1s0Z1IY7rCnsxBb6eqtStQnN5XcFatGaVmKL0NuYSOEtmmYswpgv1pd5BFBAEFH7Rp+AvBCwjxwvTcEhLxlJ+Gqy+8oJuiif7+EdkqZNN4q38w0HPERU0uMTIojwfmfuMjIUKQshTJWcmMCKJQMdADf+yRDn8X/JkzEkZq/2qK04DkktuceE61+iA2K3KzMyZXxbJVwJ2rsidaSwJHHOvbhHOgWVDJtqa2T2wSyNJVHz63hym8Lq15XVi6j/62Y2JlwB4UIYb1lFwkoFrH1N3hU2/S/d18yzUkhzD5J6DXPEHCZKWYf3X944Y6sJbZVn8ueYt5UFoYJ9K/iSEf4xX5JWsACSWttULlnZj4Q9bRjuCgsJ90/slFAz9Hykj0tP4TVBUeeA7AxT0= ceph-c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:28:06.330 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.547+0000 7f17cd4ce700 1 Processor -- start 2024-03-26T06:28:06.330 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.547+0000 7f17cd4ce700 1 -- start start 2024-03-26T06:28:06.330 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.547+0000 7f17cd4ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c8106ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:06.330 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.547+0000 7f17cd4ce700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c8107270 con 0x7f17c8106880 2024-03-26T06:28:06.330 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.547+0000 7f17c6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c8106ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:06.330 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.548+0000 7f17c6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c8106ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56920/0 (socket says 172.21.15.38:56920) 2024-03-26T06:28:06.330 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.548+0000 7f17c6ffd700 1 -- 172.21.15.38:0/2984929658 learned_addr learned my addr 172.21.15.38:0/2984929658 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:06.330 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.548+0000 7f17c6ffd700 1 -- 172.21.15.38:0/2984929658 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c8107a80 con 0x7f17c8106880 2024-03-26T06:28:06.330 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.548+0000 7f17c6ffd700 1 --2- 172.21.15.38:0/2984929658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c8106ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f17b800d180 tx=0x7f17b800d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613aed5e42816326 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:06.331 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.548+0000 7f17c5ffb700 1 -- 172.21.15.38:0/2984929658 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17b8019070 con 0x7f17c8106880 2024-03-26T06:28:06.331 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.548+0000 7f17c5ffb700 1 -- 172.21.15.38:0/2984929658 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17b800de20 con 0x7f17c8106880 2024-03-26T06:28:06.331 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.548+0000 7f17c5ffb700 1 -- 172.21.15.38:0/2984929658 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17b8009b90 con 0x7f17c8106880 2024-03-26T06:28:06.331 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.549+0000 7f17cd4ce700 1 -- 172.21.15.38:0/2984929658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 msgr2=0x7f17c8106ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:06.331 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.549+0000 7f17cd4ce700 1 --2- 172.21.15.38:0/2984929658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c8106ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f17b800d180 tx=0x7f17b800d490 comp rx=0 tx=0).stop 2024-03-26T06:28:06.331 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.549+0000 7f17cd4ce700 1 -- 172.21.15.38:0/2984929658 shutdown_connections 2024-03-26T06:28:06.331 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.549+0000 7f17cd4ce700 1 --2- 172.21.15.38:0/2984929658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c8106ca0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:06.331 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.549+0000 7f17cd4ce700 1 -- 172.21.15.38:0/2984929658 >> 172.21.15.38:0/2984929658 conn(0x7f17c8101e40 msgr2=0x7f17c8104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:06.331 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.549+0000 7f17cd4ce700 1 -- 172.21.15.38:0/2984929658 shutdown_connections 2024-03-26T06:28:06.331 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.549+0000 7f17cd4ce700 1 -- 172.21.15.38:0/2984929658 wait complete. 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.549+0000 7f17cd4ce700 1 Processor -- start 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17cd4ce700 1 -- start start 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17cd4ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c819db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17cd4ce700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c807af50 con 0x7f17c8106880 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17c6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c819db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17c6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c819db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56926/0 (socket says 172.21.15.38:56926) 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17c6ffd700 1 -- 172.21.15.38:0/4166502796 learned_addr learned my addr 172.21.15.38:0/4166502796 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17c6ffd700 1 -- 172.21.15.38:0/4166502796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c819e0d0 con 0x7f17c8106880 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17c6ffd700 1 --2- 172.21.15.38:0/4166502796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c819db90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f17b800bd10 tx=0x7f17b800dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17bffff700 1 -- 172.21.15.38:0/4166502796 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17b8019040 con 0x7f17c8106880 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17bffff700 1 -- 172.21.15.38:0/4166502796 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f17b8009e50 con 0x7f17c8106880 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17cd4ce700 1 -- 172.21.15.38:0/4166502796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c819e360 con 0x7f17c8106880 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17bffff700 1 -- 172.21.15.38:0/4166502796 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f17b801f5a0 con 0x7f17c8106880 2024-03-26T06:28:06.332 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.550+0000 7f17cd4ce700 1 -- 172.21.15.38:0/4166502796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c81a1000 con 0x7f17c8106880 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.551+0000 7f17bffff700 1 -- 172.21.15.38:0/4166502796 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f17b801f850 con 0x7f17c8106880 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.551+0000 7f17cd4ce700 1 -- 172.21.15.38:0/4166502796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c8068f20 con 0x7f17c8106880 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.552+0000 7f17bffff700 1 --2- 172.21.15.38:0/4166502796 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f17b4038f70 0x7f17b403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.552+0000 7f17bffff700 1 -- 172.21.15.38:0/4166502796 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f17b8011050 con 0x7f17c8106880 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.552+0000 7f17c67fc700 1 --2- 172.21.15.38:0/4166502796 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f17b4038f70 0x7f17b403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.552+0000 7f17c67fc700 1 --2- 172.21.15.38:0/4166502796 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f17b4038f70 0x7f17b403b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f17b0006fd0 tx=0x7f17b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.557+0000 7f17bffff700 1 -- 172.21.15.38:0/4166502796 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f17b8003ce0 con 0x7f17c8106880 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.763+0000 7f17cd4ce700 1 -- 172.21.15.38:0/4166502796 --> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f17c8062220 con 0x7f17b4038f70 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.764+0000 7f17bffff700 1 -- 172.21.15.38:0/4166502796 <== mgr.14120 v2:172.21.15.38:6800/4246298230 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f17c8062220 con 0x7f17b4038f70 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.766+0000 7f17cd4ce700 1 -- 172.21.15.38:0/4166502796 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f17b4038f70 msgr2=0x7f17b403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.766+0000 7f17cd4ce700 1 --2- 172.21.15.38:0/4166502796 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f17b4038f70 0x7f17b403b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f17b0006fd0 tx=0x7f17b0006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.766+0000 7f17cd4ce700 1 -- 172.21.15.38:0/4166502796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 msgr2=0x7f17c819db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:06.333 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.766+0000 7f17cd4ce700 1 --2- 172.21.15.38:0/4166502796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c819db90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f17b800bd10 tx=0x7f17b800dda0 comp rx=0 tx=0).stop 2024-03-26T06:28:06.334 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.766+0000 7f17cd4ce700 1 -- 172.21.15.38:0/4166502796 shutdown_connections 2024-03-26T06:28:06.334 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.766+0000 7f17cd4ce700 1 --2- 172.21.15.38:0/4166502796 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f17b4038f70 0x7f17b403b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:06.334 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.766+0000 7f17cd4ce700 1 --2- 172.21.15.38:0/4166502796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17c8106880 0x7f17c819db90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:06.334 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.766+0000 7f17cd4ce700 1 -- 172.21.15.38:0/4166502796 >> 172.21.15.38:0/4166502796 conn(0x7f17c8101e40 msgr2=0x7f17c818fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:06.334 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.766+0000 7f17cd4ce700 1 -- 172.21.15.38:0/4166502796 shutdown_connections 2024-03-26T06:28:06.334 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:05.767+0000 7f17cd4ce700 1 -- 172.21.15.38:0/4166502796 wait complete. 2024-03-26T06:28:06.334 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T06:28:06.334 INFO:teuthology.orchestra.run.smithi038.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T06:28:06.334 INFO:teuthology.orchestra.run.smithi038.stdout:Adding host smithi038... 2024-03-26T06:28:06.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:06 smithi038 ceph-mon[32265]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:08.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:07 smithi038 ceph-mon[32265]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:08.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:07 smithi038 ceph-mon[32265]: Deploying cephadm binary to smithi038 2024-03-26T06:28:11.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Added host 'smithi038' with addr '172.21.15.38' 2024-03-26T06:28:11.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.719+0000 7f754c955700 1 Processor -- start 2024-03-26T06:28:11.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.719+0000 7f754c955700 1 -- start start 2024-03-26T06:28:11.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.720+0000 7f754c955700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754810a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:11.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.720+0000 7f754c955700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754807bd70 con 0x7f75481084c0 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.720+0000 7f754659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754810a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.720+0000 7f754659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754810a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56934/0 (socket says 172.21.15.38:56934) 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.720+0000 7f754659c700 1 -- 172.21.15.38:0/1439316954 learned_addr learned my addr 172.21.15.38:0/1439316954 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.720+0000 7f754659c700 1 -- 172.21.15.38:0/1439316954 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754810adf0 con 0x7f75481084c0 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.720+0000 7f754659c700 1 --2- 172.21.15.38:0/1439316954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754810a8b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f75380098a0 tx=0x7f7538009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb241794309e1c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.720+0000 7f754559a700 1 -- 172.21.15.38:0/1439316954 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7538004160 con 0x7f75481084c0 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.720+0000 7f754559a700 1 -- 172.21.15.38:0/1439316954 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7538004300 con 0x7f75481084c0 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.720+0000 7f754559a700 1 -- 172.21.15.38:0/1439316954 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7538011b50 con 0x7f75481084c0 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.721+0000 7f754c955700 1 -- 172.21.15.38:0/1439316954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 msgr2=0x7f754810a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.721+0000 7f754c955700 1 --2- 172.21.15.38:0/1439316954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754810a8b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f75380098a0 tx=0x7f7538009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.721+0000 7f754c955700 1 -- 172.21.15.38:0/1439316954 shutdown_connections 2024-03-26T06:28:11.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.721+0000 7f754c955700 1 --2- 172.21.15.38:0/1439316954 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754810a8b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.721+0000 7f754c955700 1 -- 172.21.15.38:0/1439316954 >> 172.21.15.38:0/1439316954 conn(0x7f7548101e10 msgr2=0x7f7548104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.721+0000 7f754c955700 1 -- 172.21.15.38:0/1439316954 shutdown_connections 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.721+0000 7f754c955700 1 -- 172.21.15.38:0/1439316954 wait complete. 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.722+0000 7f754c955700 1 Processor -- start 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.722+0000 7f754c955700 1 -- start start 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.722+0000 7f754c955700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754819b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.722+0000 7f754c955700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754810bd60 con 0x7f75481084c0 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.722+0000 7f754659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754819b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.722+0000 7f754659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754819b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56944/0 (socket says 172.21.15.38:56944) 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.722+0000 7f754659c700 1 -- 172.21.15.38:0/2666073079 learned_addr learned my addr 172.21.15.38:0/2666073079 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.723+0000 7f754659c700 1 -- 172.21.15.38:0/2666073079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754819be10 con 0x7f75481084c0 2024-03-26T06:28:11.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.723+0000 7f754659c700 1 --2- 172.21.15.38:0/2666073079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754819b8d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7538011b60 tx=0x7f7538011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.723+0000 7f75377fe700 1 -- 172.21.15.38:0/2666073079 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7538003b20 con 0x7f75481084c0 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.723+0000 7f75377fe700 1 -- 172.21.15.38:0/2666073079 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7538003cc0 con 0x7f75481084c0 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.723+0000 7f754c955700 1 -- 172.21.15.38:0/2666073079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754819c0a0 con 0x7f75481084c0 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.723+0000 7f75377fe700 1 -- 172.21.15.38:0/2666073079 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7538024840 con 0x7f75481084c0 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.723+0000 7f754c955700 1 -- 172.21.15.38:0/2666073079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754819ed90 con 0x7f75481084c0 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.724+0000 7f75377fe700 1 -- 172.21.15.38:0/2666073079 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7538022020 con 0x7f75481084c0 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.724+0000 7f754c955700 1 -- 172.21.15.38:0/2666073079 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7548063410 con 0x7f75481084c0 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.724+0000 7f75377fe700 1 --2- 172.21.15.38:0/2666073079 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f7530038f70 0x7f753003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.724+0000 7f75377fe700 1 -- 172.21.15.38:0/2666073079 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7538011e00 con 0x7f75481084c0 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.724+0000 7f7545d9b700 1 --2- 172.21.15.38:0/2666073079 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f7530038f70 0x7f753003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.725+0000 7f7545d9b700 1 --2- 172.21.15.38:0/2666073079 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f7530038f70 0x7f753003b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f753c006fd0 tx=0x7f753c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:11.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.729+0000 7f75377fe700 1 -- 172.21.15.38:0/2666073079 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f753800b3e0 con 0x7f75481084c0 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:06.938+0000 7f754c955700 1 -- 172.21.15.38:0/2666073079 --> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}) v1 -- 0x7f75481030d0 con 0x7f7530038f70 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:11.311+0000 7f75377fe700 1 -- 172.21.15.38:0/2666073079 <== mgr.14120 v2:172.21.15.38:6800/4246298230 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f75481030d0 con 0x7f7530038f70 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:11.314+0000 7f754c955700 1 -- 172.21.15.38:0/2666073079 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f7530038f70 msgr2=0x7f753003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:11.314+0000 7f754c955700 1 --2- 172.21.15.38:0/2666073079 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f7530038f70 0x7f753003b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f753c006fd0 tx=0x7f753c006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:11.314+0000 7f754c955700 1 -- 172.21.15.38:0/2666073079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 msgr2=0x7f754819b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:11.314+0000 7f754c955700 1 --2- 172.21.15.38:0/2666073079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754819b8d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7538011b60 tx=0x7f7538011c40 comp rx=0 tx=0).stop 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:11.314+0000 7f754c955700 1 -- 172.21.15.38:0/2666073079 shutdown_connections 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:11.314+0000 7f754c955700 1 --2- 172.21.15.38:0/2666073079 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f7530038f70 0x7f753003b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:11.314+0000 7f754c955700 1 --2- 172.21.15.38:0/2666073079 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75481084c0 0x7f754819b8d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:11.314+0000 7f754c955700 1 -- 172.21.15.38:0/2666073079 >> 172.21.15.38:0/2666073079 conn(0x7f7548101e10 msgr2=0x7f75481029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:11.314+0000 7f754c955700 1 -- 172.21.15.38:0/2666073079 shutdown_connections 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:11.314+0000 7f754c955700 1 -- 172.21.15.38:0/2666073079 wait complete. 2024-03-26T06:28:11.917 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mon service... 2024-03-26T06:28:12.430 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:12 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:12.430 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:12 smithi038 ceph-mon[32265]: Added host smithi038 2024-03-26T06:28:12.431 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:12 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:13.382 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.428+0000 7feb28f1b700 1 Processor -- start 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.428+0000 7feb28f1b700 1 -- start start 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.428+0000 7feb28f1b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2410a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.428+0000 7feb28f1b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2407bdb0 con 0x7feb24108430 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.428+0000 7feb2259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2410a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.428+0000 7feb2259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2410a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56946/0 (socket says 172.21.15.38:56946) 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.428+0000 7feb2259c700 1 -- 172.21.15.38:0/1617054812 learned_addr learned my addr 172.21.15.38:0/1617054812 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.429+0000 7feb2259c700 1 -- 172.21.15.38:0/1617054812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2410ad60 con 0x7feb24108430 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.429+0000 7feb2259c700 1 --2- 172.21.15.38:0/1617054812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2410a820 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb140098a0 tx=0x7feb14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a0821f9aff12b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.429+0000 7feb2159a700 1 -- 172.21.15.38:0/1617054812 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb14004160 con 0x7feb24108430 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.429+0000 7feb2159a700 1 -- 172.21.15.38:0/1617054812 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb14004300 con 0x7feb24108430 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.429+0000 7feb2159a700 1 -- 172.21.15.38:0/1617054812 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb14011b50 con 0x7feb24108430 2024-03-26T06:28:13.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.429+0000 7feb28f1b700 1 -- 172.21.15.38:0/1617054812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 msgr2=0x7feb2410a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.429+0000 7feb28f1b700 1 --2- 172.21.15.38:0/1617054812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2410a820 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb140098a0 tx=0x7feb14009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.430+0000 7feb28f1b700 1 -- 172.21.15.38:0/1617054812 shutdown_connections 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.430+0000 7feb28f1b700 1 --2- 172.21.15.38:0/1617054812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2410a820 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.430+0000 7feb28f1b700 1 -- 172.21.15.38:0/1617054812 >> 172.21.15.38:0/1617054812 conn(0x7feb24101dc0 msgr2=0x7feb24104220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.430+0000 7feb28f1b700 1 -- 172.21.15.38:0/1617054812 shutdown_connections 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.430+0000 7feb28f1b700 1 -- 172.21.15.38:0/1617054812 wait complete. 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.430+0000 7feb28f1b700 1 Processor -- start 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.430+0000 7feb28f1b700 1 -- start start 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb28f1b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2419b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb28f1b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2410bd20 con 0x7feb24108430 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb2259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2419b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb2259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2419b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56950/0 (socket says 172.21.15.38:56950) 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb2259c700 1 -- 172.21.15.38:0/3687397444 learned_addr learned my addr 172.21.15.38:0/3687397444 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:13.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb2259c700 1 -- 172.21.15.38:0/3687397444 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2419be30 con 0x7feb24108430 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb2259c700 1 --2- 172.21.15.38:0/3687397444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2419b8f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feb14011670 tx=0x7feb14011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb137fe700 1 -- 172.21.15.38:0/3687397444 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb140039c0 con 0x7feb24108430 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb137fe700 1 -- 172.21.15.38:0/3687397444 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feb14011e50 con 0x7feb24108430 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb28f1b700 1 -- 172.21.15.38:0/3687397444 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2419c0c0 con 0x7feb24108430 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb137fe700 1 -- 172.21.15.38:0/3687397444 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feb1401a5c0 con 0x7feb24108430 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.431+0000 7feb28f1b700 1 -- 172.21.15.38:0/3687397444 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2419edb0 con 0x7feb24108430 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.432+0000 7feb28f1b700 1 -- 172.21.15.38:0/3687397444 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb24063410 con 0x7feb24108430 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.433+0000 7feb137fe700 1 -- 172.21.15.38:0/3687397444 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7feb14024430 con 0x7feb24108430 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.433+0000 7feb137fe700 1 --2- 172.21.15.38:0/3687397444 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7feb0c038ed0 0x7feb0c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.433+0000 7feb137fe700 1 -- 172.21.15.38:0/3687397444 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7feb1404d260 con 0x7feb24108430 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.433+0000 7feb21d9b700 1 --2- 172.21.15.38:0/3687397444 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7feb0c038ed0 0x7feb0c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.433+0000 7feb21d9b700 1 --2- 172.21.15.38:0/3687397444 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7feb0c038ed0 0x7feb0c03b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7feb18006fd0 tx=0x7feb18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:13.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.438+0000 7feb137fe700 1 -- 172.21.15.38:0/3687397444 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7feb14017ca0 con 0x7feb24108430 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.672+0000 7feb28f1b700 1 -- 172.21.15.38:0/3687397444 --> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7feb24103010 con 0x7feb0c038ed0 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.676+0000 7feb137fe700 1 -- 172.21.15.38:0/3687397444 <== mgr.14120 v2:172.21.15.38:6800/4246298230 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7feb24103010 con 0x7feb0c038ed0 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.678+0000 7feb28f1b700 1 -- 172.21.15.38:0/3687397444 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7feb0c038ed0 msgr2=0x7feb0c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.678+0000 7feb28f1b700 1 --2- 172.21.15.38:0/3687397444 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7feb0c038ed0 0x7feb0c03b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7feb18006fd0 tx=0x7feb18006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.678+0000 7feb28f1b700 1 -- 172.21.15.38:0/3687397444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 msgr2=0x7feb2419b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.678+0000 7feb28f1b700 1 --2- 172.21.15.38:0/3687397444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2419b8f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feb14011670 tx=0x7feb14011b60 comp rx=0 tx=0).stop 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.678+0000 7feb28f1b700 1 -- 172.21.15.38:0/3687397444 shutdown_connections 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.678+0000 7feb28f1b700 1 --2- 172.21.15.38:0/3687397444 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7feb0c038ed0 0x7feb0c03b390 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.678+0000 7feb28f1b700 1 --2- 172.21.15.38:0/3687397444 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb24108430 0x7feb2419b8f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.678+0000 7feb28f1b700 1 -- 172.21.15.38:0/3687397444 >> 172.21.15.38:0/3687397444 conn(0x7feb24101dc0 msgr2=0x7feb24102900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.678+0000 7feb28f1b700 1 -- 172.21.15.38:0/3687397444 shutdown_connections 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:12.679+0000 7feb28f1b700 1 -- 172.21.15.38:0/3687397444 wait complete. 2024-03-26T06:28:13.386 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mgr service... 2024-03-26T06:28:13.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:13 smithi038 ceph-mon[32265]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:13.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:13 smithi038 ceph-mon[32265]: Saving service mon spec with placement count:5 2024-03-26T06:28:13.937 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:13 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:14.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T06:28:14.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.768+0000 7f6655a08700 1 Processor -- start 2024-03-26T06:28:14.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.768+0000 7f6655a08700 1 -- start start 2024-03-26T06:28:14.766 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.769+0000 7f6655a08700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f6650107e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.769+0000 7f6655a08700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6650108400 con 0x7f6650107a10 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.769+0000 7f664effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f6650107e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.769+0000 7f664effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f6650107e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56960/0 (socket says 172.21.15.38:56960) 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.769+0000 7f664effd700 1 -- 172.21.15.38:0/2866110173 learned_addr learned my addr 172.21.15.38:0/2866110173 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.769+0000 7f664effd700 1 -- 172.21.15.38:0/2866110173 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6650108b70 con 0x7f6650107a10 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.769+0000 7f664effd700 1 --2- 172.21.15.38:0/2866110173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f6650107e30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f66400098a0 tx=0x7f6640009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e546b68f78a020 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.769+0000 7f664dffb700 1 -- 172.21.15.38:0/2866110173 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6640004160 con 0x7f6650107a10 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.769+0000 7f664dffb700 1 -- 172.21.15.38:0/2866110173 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6640004300 con 0x7f6650107a10 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.769+0000 7f664dffb700 1 -- 172.21.15.38:0/2866110173 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6640011b50 con 0x7f6650107a10 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.770+0000 7f6655a08700 1 -- 172.21.15.38:0/2866110173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 msgr2=0x7f6650107e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.770+0000 7f6655a08700 1 --2- 172.21.15.38:0/2866110173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f6650107e30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f66400098a0 tx=0x7f6640009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.770+0000 7f6655a08700 1 -- 172.21.15.38:0/2866110173 shutdown_connections 2024-03-26T06:28:14.767 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.770+0000 7f6655a08700 1 --2- 172.21.15.38:0/2866110173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f6650107e30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.770+0000 7f6655a08700 1 -- 172.21.15.38:0/2866110173 >> 172.21.15.38:0/2866110173 conn(0x7f6650102fb0 msgr2=0x7f66501053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.770+0000 7f6655a08700 1 -- 172.21.15.38:0/2866110173 shutdown_connections 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.770+0000 7f6655a08700 1 -- 172.21.15.38:0/2866110173 wait complete. 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.771+0000 7f6655a08700 1 Processor -- start 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.771+0000 7f6655a08700 1 -- start start 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.771+0000 7f6655a08700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f665019b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.771+0000 7f6655a08700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665010bf20 con 0x7f6650107a10 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.771+0000 7f664effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f665019b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.771+0000 7f664effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f665019b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56966/0 (socket says 172.21.15.38:56966) 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.771+0000 7f664effd700 1 -- 172.21.15.38:0/343756730 learned_addr learned my addr 172.21.15.38:0/343756730 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.771+0000 7f664effd700 1 -- 172.21.15.38:0/343756730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665019bea0 con 0x7f6650107a10 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.772+0000 7f664effd700 1 --2- 172.21.15.38:0/343756730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f665019b960 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6640011670 tx=0x7f6640011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:14.768 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.772+0000 7f6654a06700 1 -- 172.21.15.38:0/343756730 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f66400039c0 con 0x7f6650107a10 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.772+0000 7f6654a06700 1 -- 172.21.15.38:0/343756730 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6640011e50 con 0x7f6650107a10 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.772+0000 7f6655a08700 1 -- 172.21.15.38:0/343756730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665019c130 con 0x7f6650107a10 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.772+0000 7f6654a06700 1 -- 172.21.15.38:0/343756730 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f664001a5c0 con 0x7f6650107a10 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.772+0000 7f6655a08700 1 -- 172.21.15.38:0/343756730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665019ee20 con 0x7f6650107a10 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.773+0000 7f6654a06700 1 -- 172.21.15.38:0/343756730 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f6640024430 con 0x7f6650107a10 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.773+0000 7f6655a08700 1 -- 172.21.15.38:0/343756730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6650063410 con 0x7f6650107a10 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.773+0000 7f6654a06700 1 --2- 172.21.15.38:0/343756730 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f6638038f20 0x7f663803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.773+0000 7f6654a06700 1 -- 172.21.15.38:0/343756730 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f664004d2f0 con 0x7f6650107a10 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.773+0000 7f664e7fc700 1 --2- 172.21.15.38:0/343756730 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f6638038f20 0x7f663803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.773+0000 7f664e7fc700 1 --2- 172.21.15.38:0/343756730 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f6638038f20 0x7f663803b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6644006fd0 tx=0x7f6644006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.778+0000 7f6654a06700 1 -- 172.21.15.38:0/343756730 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f664002d9f0 con 0x7f6650107a10 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:13.996+0000 7f6655a08700 1 -- 172.21.15.38:0/343756730 --> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f6650104290 con 0x7f6638038f20 2024-03-26T06:28:14.769 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:14.002+0000 7f6654a06700 1 -- 172.21.15.38:0/343756730 <== mgr.14120 v2:172.21.15.38:6800/4246298230 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f6650104290 con 0x7f6638038f20 2024-03-26T06:28:14.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:14.004+0000 7f6655a08700 1 -- 172.21.15.38:0/343756730 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f6638038f20 msgr2=0x7f663803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:14.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:14.004+0000 7f6655a08700 1 --2- 172.21.15.38:0/343756730 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f6638038f20 0x7f663803b3e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6644006fd0 tx=0x7f6644006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:14.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:14.004+0000 7f6655a08700 1 -- 172.21.15.38:0/343756730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 msgr2=0x7f665019b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:14.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:14.004+0000 7f6655a08700 1 --2- 172.21.15.38:0/343756730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f665019b960 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6640011670 tx=0x7f6640011b60 comp rx=0 tx=0).stop 2024-03-26T06:28:14.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:14.005+0000 7f6655a08700 1 -- 172.21.15.38:0/343756730 shutdown_connections 2024-03-26T06:28:14.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:14.005+0000 7f6655a08700 1 --2- 172.21.15.38:0/343756730 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f6638038f20 0x7f663803b3e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:14.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:14.005+0000 7f6655a08700 1 --2- 172.21.15.38:0/343756730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6650107a10 0x7f665019b960 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:14.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:14.005+0000 7f6655a08700 1 -- 172.21.15.38:0/343756730 >> 172.21.15.38:0/343756730 conn(0x7f6650102fb0 msgr2=0x7f6650103b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:14.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:14.005+0000 7f6655a08700 1 -- 172.21.15.38:0/343756730 shutdown_connections 2024-03-26T06:28:14.770 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:14.005+0000 7f6655a08700 1 -- 172.21.15.38:0/343756730 wait complete. 2024-03-26T06:28:15.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:15 smithi038 ceph-mon[32265]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:15.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:15 smithi038 ceph-mon[32265]: Saving service mgr spec with placement count:2 2024-03-26T06:28:15.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:15 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:16.375 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.544+0000 7efea757f700 1 Processor -- start 2024-03-26T06:28:16.375 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.544+0000 7efea757f700 1 -- start start 2024-03-26T06:28:16.375 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.544+0000 7efea757f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea0108610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:16.375 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.544+0000 7efea757f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea007be30 con 0x7efea01061e0 2024-03-26T06:28:16.375 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.544+0000 7efea531b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea0108610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:16.375 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.544+0000 7efea531b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea0108610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47716/0 (socket says 172.21.15.38:47716) 2024-03-26T06:28:16.375 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.544+0000 7efea531b700 1 -- 172.21.15.38:0/2928350074 learned_addr learned my addr 172.21.15.38:0/2928350074 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.545+0000 7efea531b700 1 -- 172.21.15.38:0/2928350074 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea0108b50 con 0x7efea01061e0 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.545+0000 7efea531b700 1 --2- 172.21.15.38:0/2928350074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea0108610 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efe940098a0 tx=0x7efe94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d35c1694382fedb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.545+0000 7efe93fff700 1 -- 172.21.15.38:0/2928350074 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe94004160 con 0x7efea01061e0 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.545+0000 7efe93fff700 1 -- 172.21.15.38:0/2928350074 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe94004300 con 0x7efea01061e0 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.545+0000 7efe93fff700 1 -- 172.21.15.38:0/2928350074 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe94011b20 con 0x7efea01061e0 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.546+0000 7efea757f700 1 -- 172.21.15.38:0/2928350074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 msgr2=0x7efea0108610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.546+0000 7efea757f700 1 --2- 172.21.15.38:0/2928350074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea0108610 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7efe940098a0 tx=0x7efe94009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.546+0000 7efea757f700 1 -- 172.21.15.38:0/2928350074 shutdown_connections 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.546+0000 7efea757f700 1 --2- 172.21.15.38:0/2928350074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea0108610 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.546+0000 7efea757f700 1 -- 172.21.15.38:0/2928350074 >> 172.21.15.38:0/2928350074 conn(0x7efea0101e40 msgr2=0x7efea0104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.546+0000 7efea757f700 1 -- 172.21.15.38:0/2928350074 shutdown_connections 2024-03-26T06:28:16.376 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.546+0000 7efea757f700 1 -- 172.21.15.38:0/2928350074 wait complete. 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.546+0000 7efea757f700 1 Processor -- start 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efea757f700 1 -- start start 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efea757f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea01a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efea757f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efea0109c20 con 0x7efea01061e0 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efea531b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea01a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efea531b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea01a1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47732/0 (socket says 172.21.15.38:47732) 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efea531b700 1 -- 172.21.15.38:0/3502518906 learned_addr learned my addr 172.21.15.38:0/3502518906 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efea531b700 1 -- 172.21.15.38:0/3502518906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efea01a2460 con 0x7efea01061e0 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efea531b700 1 --2- 172.21.15.38:0/3502518906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea01a1f20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efe940116a0 tx=0x7efe94011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efe927fc700 1 -- 172.21.15.38:0/3502518906 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe94003a40 con 0x7efea01061e0 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efea757f700 1 -- 172.21.15.38:0/3502518906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efea01a26f0 con 0x7efea01061e0 2024-03-26T06:28:16.377 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efe927fc700 1 -- 172.21.15.38:0/3502518906 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe94003be0 con 0x7efea01061e0 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.547+0000 7efe927fc700 1 -- 172.21.15.38:0/3502518906 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe9401a750 con 0x7efea01061e0 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.548+0000 7efea757f700 1 -- 172.21.15.38:0/3502518906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efea01a2b50 con 0x7efea01061e0 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.548+0000 7efe927fc700 1 -- 172.21.15.38:0/3502518906 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7efe9401aa60 con 0x7efea01061e0 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.548+0000 7efe927fc700 1 --2- 172.21.15.38:0/3502518906 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7efe8c038f20 0x7efe8c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.549+0000 7efe927fc700 1 -- 172.21.15.38:0/3502518906 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efe9404d5c0 con 0x7efea01061e0 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.549+0000 7efea4b1a700 1 --2- 172.21.15.38:0/3502518906 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7efe8c038f20 0x7efe8c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.549+0000 7efea757f700 1 -- 172.21.15.38:0/3502518906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea0068f20 con 0x7efea01061e0 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.549+0000 7efea4b1a700 1 --2- 172.21.15.38:0/3502518906 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7efe8c038f20 0x7efe8c03b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe9c006fd0 tx=0x7efe9c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.555+0000 7efe927fc700 1 -- 172.21.15.38:0/3502518906 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7efe94011e20 con 0x7efea01061e0 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.786+0000 7efea757f700 1 -- 172.21.15.38:0/3502518906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7efea002da80 con 0x7efea01061e0 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.790+0000 7efe927fc700 1 -- 172.21.15.38:0/3502518906 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7efe9400bc40 con 0x7efea01061e0 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.792+0000 7efea757f700 1 -- 172.21.15.38:0/3502518906 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7efe8c038f20 msgr2=0x7efe8c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:16.378 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.792+0000 7efea757f700 1 --2- 172.21.15.38:0/3502518906 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7efe8c038f20 0x7efe8c03b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7efe9c006fd0 tx=0x7efe9c006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:16.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.792+0000 7efea757f700 1 -- 172.21.15.38:0/3502518906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 msgr2=0x7efea01a1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:16.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.792+0000 7efea757f700 1 --2- 172.21.15.38:0/3502518906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea01a1f20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7efe940116a0 tx=0x7efe94011ba0 comp rx=0 tx=0).stop 2024-03-26T06:28:16.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.792+0000 7efea757f700 1 -- 172.21.15.38:0/3502518906 shutdown_connections 2024-03-26T06:28:16.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.792+0000 7efea757f700 1 --2- 172.21.15.38:0/3502518906 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7efe8c038f20 0x7efe8c03b3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:16.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.792+0000 7efea757f700 1 --2- 172.21.15.38:0/3502518906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efea01061e0 0x7efea01a1f20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:16.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.792+0000 7efea757f700 1 -- 172.21.15.38:0/3502518906 >> 172.21.15.38:0/3502518906 conn(0x7efea0101e40 msgr2=0x7efea0104230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:16.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.792+0000 7efea757f700 1 -- 172.21.15.38:0/3502518906 shutdown_connections 2024-03-26T06:28:16.379 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:15.792+0000 7efea757f700 1 -- 172.21.15.38:0/3502518906 wait complete. 2024-03-26T06:28:16.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:16 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:16.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:16 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3502518906' entity='client.admin' 2024-03-26T06:28:16.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:16 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:17.792 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.829+0000 7f74d4511700 1 Processor -- start 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.829+0000 7f74d4511700 1 -- start start 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.829+0000 7f74d4511700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc106cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.830+0000 7f74d4511700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc107280 con 0x7f74cc106890 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.830+0000 7f74d22ad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc106cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.830+0000 7f74d22ad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc106cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47736/0 (socket says 172.21.15.38:47736) 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.830+0000 7f74d22ad700 1 -- 172.21.15.38:0/1759799523 learned_addr learned my addr 172.21.15.38:0/1759799523 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.830+0000 7f74d22ad700 1 -- 172.21.15.38:0/1759799523 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc107a90 con 0x7f74cc106890 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.830+0000 7f74d22ad700 1 --2- 172.21.15.38:0/1759799523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc106cb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f74c80098a0 tx=0x7f74c8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b686b43ab12d1279 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.830+0000 7f74d12ab700 1 -- 172.21.15.38:0/1759799523 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74c8004160 con 0x7f74cc106890 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.830+0000 7f74d12ab700 1 -- 172.21.15.38:0/1759799523 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74c8004300 con 0x7f74cc106890 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.830+0000 7f74d12ab700 1 -- 172.21.15.38:0/1759799523 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74c8011b20 con 0x7f74cc106890 2024-03-26T06:28:17.793 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.831+0000 7f74d4511700 1 -- 172.21.15.38:0/1759799523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 msgr2=0x7f74cc106cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.831+0000 7f74d4511700 1 --2- 172.21.15.38:0/1759799523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc106cb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f74c80098a0 tx=0x7f74c8009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.831+0000 7f74d4511700 1 -- 172.21.15.38:0/1759799523 shutdown_connections 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.831+0000 7f74d4511700 1 --2- 172.21.15.38:0/1759799523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc106cb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.831+0000 7f74d4511700 1 -- 172.21.15.38:0/1759799523 >> 172.21.15.38:0/1759799523 conn(0x7f74cc101eb0 msgr2=0x7f74cc1042d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.831+0000 7f74d4511700 1 -- 172.21.15.38:0/1759799523 shutdown_connections 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.831+0000 7f74d4511700 1 -- 172.21.15.38:0/1759799523 wait complete. 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.832+0000 7f74d4511700 1 Processor -- start 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.832+0000 7f74d4511700 1 -- start start 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.832+0000 7f74d4511700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc19b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.832+0000 7f74d4511700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74cc10ae40 con 0x7f74cc106890 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.832+0000 7f74d22ad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc19b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.832+0000 7f74d22ad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc19b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47748/0 (socket says 172.21.15.38:47748) 2024-03-26T06:28:17.794 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.832+0000 7f74d22ad700 1 -- 172.21.15.38:0/89010735 learned_addr learned my addr 172.21.15.38:0/89010735 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.832+0000 7f74d22ad700 1 -- 172.21.15.38:0/89010735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74cc19be40 con 0x7f74cc106890 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.832+0000 7f74d22ad700 1 --2- 172.21.15.38:0/89010735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc19b900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f74c80117e0 tx=0x7f74c8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.832+0000 7f74c37fe700 1 -- 172.21.15.38:0/89010735 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74c8003bf0 con 0x7f74cc106890 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.833+0000 7f74c37fe700 1 -- 172.21.15.38:0/89010735 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74c801a910 con 0x7f74cc106890 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.833+0000 7f74d4511700 1 -- 172.21.15.38:0/89010735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74cc19c0d0 con 0x7f74cc106890 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.833+0000 7f74c37fe700 1 -- 172.21.15.38:0/89010735 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f74c80249a0 con 0x7f74cc106890 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.833+0000 7f74d4511700 1 -- 172.21.15.38:0/89010735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74cc19c530 con 0x7f74cc106890 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.833+0000 7f74c37fe700 1 -- 172.21.15.38:0/89010735 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f74c8022020 con 0x7f74cc106890 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.833+0000 7f74d4511700 1 -- 172.21.15.38:0/89010735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74cc063410 con 0x7f74cc106890 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.834+0000 7f74c37fe700 1 --2- 172.21.15.38:0/89010735 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f74b8038f20 0x7f74b803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.834+0000 7f74c37fe700 1 -- 172.21.15.38:0/89010735 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f74c804d2b0 con 0x7f74cc106890 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.834+0000 7f74d1aac700 1 --2- 172.21.15.38:0/89010735 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f74b8038f20 0x7f74b803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.834+0000 7f74d1aac700 1 --2- 172.21.15.38:0/89010735 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f74b8038f20 0x7f74b803b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f74bc006fd0 tx=0x7f74bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:17.795 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:16.839+0000 7f74c37fe700 1 -- 172.21.15.38:0/89010735 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f74c8011e20 con 0x7f74cc106890 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.064+0000 7f74d4511700 1 -- 172.21.15.38:0/89010735 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f74cc0629f0 con 0x7f74cc106890 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.067+0000 7f74c37fe700 1 -- 172.21.15.38:0/89010735 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f74c8019c70 con 0x7f74cc106890 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.069+0000 7f74d4511700 1 -- 172.21.15.38:0/89010735 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f74b8038f20 msgr2=0x7f74b803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.069+0000 7f74d4511700 1 --2- 172.21.15.38:0/89010735 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f74b8038f20 0x7f74b803b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f74bc006fd0 tx=0x7f74bc006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.069+0000 7f74d4511700 1 -- 172.21.15.38:0/89010735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 msgr2=0x7f74cc19b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.069+0000 7f74d4511700 1 --2- 172.21.15.38:0/89010735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc19b900 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f74c80117e0 tx=0x7f74c8011c10 comp rx=0 tx=0).stop 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.069+0000 7f74d4511700 1 -- 172.21.15.38:0/89010735 shutdown_connections 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.069+0000 7f74d4511700 1 --2- 172.21.15.38:0/89010735 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7f74b8038f20 0x7f74b803b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.069+0000 7f74d4511700 1 --2- 172.21.15.38:0/89010735 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74cc106890 0x7f74cc19b900 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.069+0000 7f74d4511700 1 -- 172.21.15.38:0/89010735 >> 172.21.15.38:0/89010735 conn(0x7f74cc101eb0 msgr2=0x7f74cc102d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.069+0000 7f74d4511700 1 -- 172.21.15.38:0/89010735 shutdown_connections 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:17.069+0000 7f74d4511700 1 -- 172.21.15.38:0/89010735 wait complete. 2024-03-26T06:28:17.796 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling the dashboard module... 2024-03-26T06:28:18.179 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:18 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/89010735' entity='client.admin' 2024-03-26T06:28:19.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:19 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2523947513' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T06:28:19.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:19 smithi038 ceph-mon[32265]: from='mgr.14120 172.21.15.38:0/1338537770' entity='mgr.x' 2024-03-26T06:28:19.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: ignoring --setuser ceph since I am not root 2024-03-26T06:28:19.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: ignoring --setgroup ceph since I am not root 2024-03-26T06:28:19.676 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:19.660+0000 7f0ffc6ce200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T06:28:19.686 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.219+0000 7fd514530700 1 Processor -- start 2024-03-26T06:28:19.686 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.219+0000 7fd514530700 1 -- start start 2024-03-26T06:28:19.686 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.219+0000 7fd514530700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c07c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:19.686 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.219+0000 7fd514530700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c07c7c0 con 0x7fd50c07dd80 2024-03-26T06:28:19.686 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.219+0000 7fd5122cc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c07c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:19.686 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.219+0000 7fd5122cc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c07c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47762/0 (socket says 172.21.15.38:47762) 2024-03-26T06:28:19.686 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.219+0000 7fd5122cc700 1 -- 172.21.15.38:0/1930771669 learned_addr learned my addr 172.21.15.38:0/1930771669 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:19.686 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.220+0000 7fd5122cc700 1 -- 172.21.15.38:0/1930771669 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c07c940 con 0x7fd50c07dd80 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.220+0000 7fd5122cc700 1 --2- 172.21.15.38:0/1930771669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c07c280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd5080098a0 tx=0x7fd508009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43284748061fbe4c server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.220+0000 7fd5112ca700 1 -- 172.21.15.38:0/1930771669 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd508004160 con 0x7fd50c07dd80 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.220+0000 7fd5112ca700 1 -- 172.21.15.38:0/1930771669 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd508004300 con 0x7fd50c07dd80 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.220+0000 7fd5112ca700 1 -- 172.21.15.38:0/1930771669 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd508011b20 con 0x7fd50c07dd80 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.221+0000 7fd514530700 1 -- 172.21.15.38:0/1930771669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 msgr2=0x7fd50c07c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.221+0000 7fd514530700 1 --2- 172.21.15.38:0/1930771669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c07c280 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd5080098a0 tx=0x7fd508009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.221+0000 7fd514530700 1 -- 172.21.15.38:0/1930771669 shutdown_connections 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.221+0000 7fd514530700 1 --2- 172.21.15.38:0/1930771669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c07c280 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.221+0000 7fd514530700 1 -- 172.21.15.38:0/1930771669 >> 172.21.15.38:0/1930771669 conn(0x7fd50c102fb0 msgr2=0x7fd50c1053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.221+0000 7fd514530700 1 -- 172.21.15.38:0/1930771669 shutdown_connections 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.221+0000 7fd514530700 1 -- 172.21.15.38:0/1930771669 wait complete. 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.221+0000 7fd514530700 1 Processor -- start 2024-03-26T06:28:19.687 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.221+0000 7fd514530700 1 -- start start 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd514530700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c19da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd514530700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd50c10dfb0 con 0x7fd50c07dd80 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd5122cc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c19da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd5122cc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c19da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47778/0 (socket says 172.21.15.38:47778) 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd5122cc700 1 -- 172.21.15.38:0/2523947513 learned_addr learned my addr 172.21.15.38:0/2523947513 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd5122cc700 1 -- 172.21.15.38:0/2523947513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c19dfd0 con 0x7fd50c07dd80 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd5122cc700 1 --2- 172.21.15.38:0/2523947513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c19da90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd5080117e0 tx=0x7fd508011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd5037fe700 1 -- 172.21.15.38:0/2523947513 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd508003bf0 con 0x7fd50c07dd80 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd5037fe700 1 -- 172.21.15.38:0/2523947513 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd50801a910 con 0x7fd50c07dd80 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd514530700 1 -- 172.21.15.38:0/2523947513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50c19e260 con 0x7fd50c07dd80 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd5037fe700 1 -- 172.21.15.38:0/2523947513 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd508024970 con 0x7fd50c07dd80 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.222+0000 7fd514530700 1 -- 172.21.15.38:0/2523947513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd50c19e6c0 con 0x7fd50c07dd80 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.223+0000 7fd5037fe700 1 -- 172.21.15.38:0/2523947513 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fd508022020 con 0x7fd50c07dd80 2024-03-26T06:28:19.688 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.223+0000 7fd514530700 1 -- 172.21.15.38:0/2523947513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50c063410 con 0x7fd50c07dd80 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.223+0000 7fd5037fe700 1 --2- 172.21.15.38:0/2523947513 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd4f8038f20 0x7fd4f803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.223+0000 7fd5037fe700 1 -- 172.21.15.38:0/2523947513 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd50804d2e0 con 0x7fd50c07dd80 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.224+0000 7fd511acb700 1 --2- 172.21.15.38:0/2523947513 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd4f8038f20 0x7fd4f803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.224+0000 7fd511acb700 1 --2- 172.21.15.38:0/2523947513 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd4f8038f20 0x7fd4f803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc006fd0 tx=0x7fd4fc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.228+0000 7fd5037fe700 1 -- 172.21.15.38:0/2523947513 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd508011e20 con 0x7fd50c07dd80 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:18.511+0000 7fd514530700 1 -- 172.21.15.38:0/2523947513 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fd50c1a1150 con 0x7fd50c07dd80 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.076+0000 7fd5037fe700 1 -- 172.21.15.38:0/2523947513 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fd508019c70 con 0x7fd50c07dd80 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.078+0000 7fd5037fe700 1 -- 172.21.15.38:0/2523947513 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fd50802d910 con 0x7fd50c07dd80 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.079+0000 7fd514530700 1 -- 172.21.15.38:0/2523947513 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd4f8038f20 msgr2=0x7fd4f803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.079+0000 7fd514530700 1 --2- 172.21.15.38:0/2523947513 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd4f8038f20 0x7fd4f803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc006fd0 tx=0x7fd4fc006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.079+0000 7fd514530700 1 -- 172.21.15.38:0/2523947513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 msgr2=0x7fd50c19da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.079+0000 7fd514530700 1 --2- 172.21.15.38:0/2523947513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c19da90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd5080117e0 tx=0x7fd508011c10 comp rx=0 tx=0).stop 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.080+0000 7fd514530700 1 -- 172.21.15.38:0/2523947513 shutdown_connections 2024-03-26T06:28:19.689 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.080+0000 7fd514530700 1 --2- 172.21.15.38:0/2523947513 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd4f8038f20 0x7fd4f803b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:19.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.080+0000 7fd514530700 1 --2- 172.21.15.38:0/2523947513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd50c07dd80 0x7fd50c19da90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:19.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.080+0000 7fd514530700 1 -- 172.21.15.38:0/2523947513 >> 172.21.15.38:0/2523947513 conn(0x7fd50c102fb0 msgr2=0x7fd50c103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:19.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.080+0000 7fd514530700 1 -- 172.21.15.38:0/2523947513 shutdown_connections 2024-03-26T06:28:19.690 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:19.080+0000 7fd514530700 1 -- 172.21.15.38:0/2523947513 wait complete. 2024-03-26T06:28:19.941 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:19.761+0000 7f0ffc6ce200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T06:28:20.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:20 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2523947513' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T06:28:20.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:20 smithi038 ceph-mon[32265]: mgrmap e9: x(active, since 19s) 2024-03-26T06:28:20.676 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:20 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:20.433+0000 7f0ffc6ce200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T06:28:20.676 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:20 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:20.529+0000 7f0ffc6ce200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T06:28:20.677 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:20 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:20.675+0000 7f0ffc6ce200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T06:28:21.025 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:20 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:20.776+0000 7f0ffc6ce200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T06:28:21.025 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:20 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:20.862+0000 7f0ffc6ce200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T06:28:21.034 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T06:28:21.034 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T06:28:21.034 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T06:28:21.034 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-26T06:28:21.034 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T06:28:21.034 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T06:28:21.034 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.130+0000 7fd955d5f700 1 Processor -- start 2024-03-26T06:28:21.034 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.131+0000 7fd955d5f700 1 -- start start 2024-03-26T06:28:21.034 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.131+0000 7fd955d5f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd950107db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:21.034 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.131+0000 7fd955d5f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd950108380 con 0x7fd950107990 2024-03-26T06:28:21.035 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.131+0000 7fd94f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd950107db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:21.035 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.131+0000 7fd94f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd950107db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47804/0 (socket says 172.21.15.38:47804) 2024-03-26T06:28:21.035 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.131+0000 7fd94f7fe700 1 -- 172.21.15.38:0/2467366217 learned_addr learned my addr 172.21.15.38:0/2467366217 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:21.035 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.131+0000 7fd94f7fe700 1 -- 172.21.15.38:0/2467366217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd950108b90 con 0x7fd950107990 2024-03-26T06:28:21.035 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.131+0000 7fd94f7fe700 1 --2- 172.21.15.38:0/2467366217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd950107db0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9400098a0 tx=0x7fd940009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de473923b2d764e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:21.035 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.131+0000 7fd94e7fc700 1 -- 172.21.15.38:0/2467366217 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd940004160 con 0x7fd950107990 2024-03-26T06:28:21.035 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.132+0000 7fd94e7fc700 1 -- 172.21.15.38:0/2467366217 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd940004300 con 0x7fd950107990 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.132+0000 7fd94e7fc700 1 -- 172.21.15.38:0/2467366217 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd940011b20 con 0x7fd950107990 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.132+0000 7fd955d5f700 1 -- 172.21.15.38:0/2467366217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 msgr2=0x7fd950107db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.132+0000 7fd955d5f700 1 --2- 172.21.15.38:0/2467366217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd950107db0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd9400098a0 tx=0x7fd940009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.132+0000 7fd955d5f700 1 -- 172.21.15.38:0/2467366217 shutdown_connections 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.132+0000 7fd955d5f700 1 --2- 172.21.15.38:0/2467366217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd950107db0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.132+0000 7fd955d5f700 1 -- 172.21.15.38:0/2467366217 >> 172.21.15.38:0/2467366217 conn(0x7fd950102f70 msgr2=0x7fd950105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.132+0000 7fd955d5f700 1 -- 172.21.15.38:0/2467366217 shutdown_connections 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.132+0000 7fd955d5f700 1 -- 172.21.15.38:0/2467366217 wait complete. 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.133+0000 7fd955d5f700 1 Processor -- start 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.133+0000 7fd955d5f700 1 -- start start 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.133+0000 7fd955d5f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd95007daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:21.036 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.133+0000 7fd955d5f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95010bf40 con 0x7fd950107990 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.133+0000 7fd94f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd95007daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.133+0000 7fd94f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd95007daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47806/0 (socket says 172.21.15.38:47806) 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.133+0000 7fd94f7fe700 1 -- 172.21.15.38:0/124053113 learned_addr learned my addr 172.21.15.38:0/124053113 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.134+0000 7fd94f7fe700 1 -- 172.21.15.38:0/124053113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95007e030 con 0x7fd950107990 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.134+0000 7fd94f7fe700 1 --2- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd95007daf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd9400116a0 tx=0x7fd940011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.134+0000 7fd94cff9700 1 -- 172.21.15.38:0/124053113 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd940003a40 con 0x7fd950107990 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.134+0000 7fd94cff9700 1 -- 172.21.15.38:0/124053113 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd940003be0 con 0x7fd950107990 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.134+0000 7fd94cff9700 1 -- 172.21.15.38:0/124053113 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd94001a720 con 0x7fd950107990 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.134+0000 7fd955d5f700 1 -- 172.21.15.38:0/124053113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95007c220 con 0x7fd950107990 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.134+0000 7fd955d5f700 1 -- 172.21.15.38:0/124053113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95007c6e0 con 0x7fd950107990 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.135+0000 7fd94cff9700 1 -- 172.21.15.38:0/124053113 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fd94001aa20 con 0x7fd950107990 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.135+0000 7fd955d5f700 1 -- 172.21.15.38:0/124053113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd950068f20 con 0x7fd950107990 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.135+0000 7fd94cff9700 1 --2- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd93c038f20 0x7fd93c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:21.037 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.135+0000 7fd94cff9700 1 -- 172.21.15.38:0/124053113 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd94004d530 con 0x7fd950107990 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.135+0000 7fd94effd700 1 -- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd93c038f20 msgr2=0x7fd93c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4246298230 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.135+0000 7fd94effd700 1 --2- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd93c038f20 0x7fd93c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.140+0000 7fd94cff9700 1 -- 172.21.15.38:0/124053113 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd940011e20 con 0x7fd950107990 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.336+0000 7fd94effd700 1 -- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd93c038f20 msgr2=0x7fd93c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4246298230 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.336+0000 7fd94effd700 1 --2- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd93c038f20 0x7fd93c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.408+0000 7fd955d5f700 1 -- 172.21.15.38:0/124053113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd95007cf80 con 0x7fd950107990 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.408+0000 7fd94cff9700 1 -- 172.21.15.38:0/124053113 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fd940019c70 con 0x7fd950107990 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.410+0000 7fd955d5f700 1 -- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd93c038f20 msgr2=0x7fd93c03b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.410+0000 7fd955d5f700 1 --2- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd93c038f20 0x7fd93c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.410+0000 7fd955d5f700 1 -- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 msgr2=0x7fd95007daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.410+0000 7fd955d5f700 1 --2- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd95007daf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd9400116a0 tx=0x7fd940011ba0 comp rx=0 tx=0).stop 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.410+0000 7fd955d5f700 1 -- 172.21.15.38:0/124053113 shutdown_connections 2024-03-26T06:28:21.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.410+0000 7fd955d5f700 1 --2- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fd93c038f20 0x7fd93c03b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:21.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.410+0000 7fd955d5f700 1 --2- 172.21.15.38:0/124053113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd950107990 0x7fd95007daf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:21.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.410+0000 7fd955d5f700 1 -- 172.21.15.38:0/124053113 >> 172.21.15.38:0/124053113 conn(0x7fd950102f70 msgr2=0x7fd950103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:21.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.410+0000 7fd955d5f700 1 -- 172.21.15.38:0/124053113 shutdown_connections 2024-03-26T06:28:21.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:20.410+0000 7fd955d5f700 1 -- 172.21.15.38:0/124053113 wait complete. 2024-03-26T06:28:21.039 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2024-03-26T06:28:21.039 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 9... 2024-03-26T06:28:21.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:21 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/124053113' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T06:28:21.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:21 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:21.135+0000 7f0ffc6ce200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T06:28:21.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:21 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:21.250+0000 7f0ffc6ce200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T06:28:21.623 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:21 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:21.341+0000 7f0ffc6ce200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T06:28:21.623 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:21 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:21.441+0000 7f0ffc6ce200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T06:28:21.624 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:21 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:21.621+0000 7f0ffc6ce200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T06:28:22.882 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:22 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:22.609+0000 7f0ffc6ce200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T06:28:22.883 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:22 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:22.743+0000 7f0ffc6ce200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T06:28:23.176 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:23 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:23.081+0000 7f0ffc6ce200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T06:28:23.176 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:23 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:23.174+0000 7f0ffc6ce200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T06:28:23.565 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:23 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:23.374+0000 7f0ffc6ce200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:28:23.565 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:23 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:23.563+0000 7f0ffc6ce200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T06:28:24.119 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:23 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:23.855+0000 7f0ffc6ce200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:28:24.119 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:24 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:24.117+0000 7f0ffc6ce200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T06:28:24.832 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:24 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:24.367+0000 7f0ffc6ce200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T06:28:25.582 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:25 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:25.214+0000 7f0ffc6ce200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T06:28:26.082 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:25 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:25.801+0000 7f0ffc6ce200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T06:28:26.082 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:25 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:25.901+0000 7f0ffc6ce200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T06:28:26.582 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:28:26.123+0000 7f0ffc6ce200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T06:28:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-mon[32265]: Active manager daemon x restarted 2024-03-26T06:28:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-mon[32265]: Activating manager daemon x 2024-03-26T06:28:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-mon[32265]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T06:28:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-mon[32265]: mgrmap e10: x(active, starting, since 0.00540145s) 2024-03-26T06:28:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:28:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T06:28:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T06:28:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T06:28:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T06:28:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-mon[32265]: Manager daemon x is now available 2024-03-26T06:28:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:27.466 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:27.466 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T06:28:27.466 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T06:28:27.466 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:27 smithi038 ceph-mon[32265]: [26/Mar/2024:06:28:26] ENGINE Bus STARTING 2024-03-26T06:28:27.466 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:27 smithi038 ceph-mon[32265]: [26/Mar/2024:06:28:26] ENGINE Serving on http://172.21.15.38:8765 2024-03-26T06:28:27.466 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:27 smithi038 ceph-mon[32265]: [26/Mar/2024:06:28:26] ENGINE Serving on https://172.21.15.38:7150 2024-03-26T06:28:27.467 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:27 smithi038 ceph-mon[32265]: [26/Mar/2024:06:28:26] ENGINE Bus STARTED 2024-03-26T06:28:27.467 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:27 smithi038 ceph-mon[32265]: mgrmap e11: x(active, since 1.00619s) 2024-03-26T06:28:27.819 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2024-03-26T06:28:27.827 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T06:28:27.827 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T06:28:27.827 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2024-03-26T06:28:27.827 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.473+0000 7fbb8cbd6700 1 Processor -- start 2024-03-26T06:28:27.828 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.473+0000 7fbb8cbd6700 1 -- start start 2024-03-26T06:28:27.828 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.473+0000 7fbb8cbd6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8810a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:27.828 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.473+0000 7fbb8cbd6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8807be10 con 0x7fbb88108420 2024-03-26T06:28:27.828 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.473+0000 7fbb8659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8810a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:27.828 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.473+0000 7fbb8659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8810a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47818/0 (socket says 172.21.15.38:47818) 2024-03-26T06:28:27.828 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.473+0000 7fbb8659c700 1 -- 172.21.15.38:0/81301114 learned_addr learned my addr 172.21.15.38:0/81301114 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:27.828 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.474+0000 7fbb8659c700 1 -- 172.21.15.38:0/81301114 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8810ad50 con 0x7fbb88108420 2024-03-26T06:28:27.828 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.474+0000 7fbb8659c700 1 --2- 172.21.15.38:0/81301114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8810a810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb780098a0 tx=0x7fbb78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12efc47f2adc1ec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:27.828 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.474+0000 7fbb8559a700 1 -- 172.21.15.38:0/81301114 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb78004160 con 0x7fbb88108420 2024-03-26T06:28:27.828 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.474+0000 7fbb8559a700 1 -- 172.21.15.38:0/81301114 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb78004300 con 0x7fbb88108420 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.474+0000 7fbb8559a700 1 -- 172.21.15.38:0/81301114 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb78011b20 con 0x7fbb88108420 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.474+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/81301114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 msgr2=0x7fbb8810a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.474+0000 7fbb8cbd6700 1 --2- 172.21.15.38:0/81301114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8810a810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbb780098a0 tx=0x7fbb78009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.475+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/81301114 shutdown_connections 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.475+0000 7fbb8cbd6700 1 --2- 172.21.15.38:0/81301114 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8810a810 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.475+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/81301114 >> 172.21.15.38:0/81301114 conn(0x7fbb88101e10 msgr2=0x7fbb88104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.475+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/81301114 shutdown_connections 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.475+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/81301114 wait complete. 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.475+0000 7fbb8cbd6700 1 Processor -- start 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.475+0000 7fbb8cbd6700 1 -- start start 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb8cbd6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8819b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:27.829 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb8cbd6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8810bd50 con 0x7fbb88108420 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb8659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8819b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb8659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8819b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47834/0 (socket says 172.21.15.38:47834) 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb8659c700 1 -- 172.21.15.38:0/3667275900 learned_addr learned my addr 172.21.15.38:0/3667275900 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb8659c700 1 -- 172.21.15.38:0/3667275900 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8819be20 con 0x7fbb88108420 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb8659c700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8819b8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbb780116a0 tx=0x7fbb78011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb78003a40 con 0x7fbb88108420 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb78003be0 con 0x7fbb88108420 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/3667275900 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb8819c110 con 0x7fbb88108420 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbb7801a720 con 0x7fbb88108420 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.476+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/3667275900 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8819eda0 con 0x7fbb88108420 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.477+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fbb7801aa80 con 0x7fbb88108420 2024-03-26T06:28:27.830 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.478+0000 7fbb777fe700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 0x7fbb7003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:27.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.478+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 --> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb7003baf0 con 0x7fbb70038f20 2024-03-26T06:28:27.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.478+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fbb7804d500 con 0x7fbb88108420 2024-03-26T06:28:27.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.478+0000 7fbb85d9b700 1 -- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 msgr2=0x7fbb7003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4246298230 2024-03-26T06:28:27.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.478+0000 7fbb85d9b700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 0x7fbb7003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T06:28:27.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.679+0000 7fbb85d9b700 1 -- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 msgr2=0x7fbb7003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4246298230 2024-03-26T06:28:27.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:21.679+0000 7fbb85d9b700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 0x7fbb7003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T06:28:27.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:22.080+0000 7fbb85d9b700 1 -- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 msgr2=0x7fbb7003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4246298230 2024-03-26T06:28:27.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:22.080+0000 7fbb85d9b700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 0x7fbb7003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T06:28:27.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:22.881+0000 7fbb85d9b700 1 -- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 msgr2=0x7fbb7003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4246298230 2024-03-26T06:28:27.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:22.881+0000 7fbb85d9b700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 0x7fbb7003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T06:28:27.831 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:24.482+0000 7fbb85d9b700 1 -- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 msgr2=0x7fbb7003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/4246298230 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:24.482+0000 7fbb85d9b700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 0x7fbb7003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:26.141+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7fbb7802d910 con 0x7fbb88108420 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:26.141+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 msgr2=0x7fbb7003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:26.141+0000 7fbb777fe700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 0x7fbb7003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.140+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7fbb7804f160 con 0x7fbb88108420 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.140+0000 7fbb777fe700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbb7003c9a0 0x7fbb7003ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.140+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbb7800fec0 con 0x7fbb7003c9a0 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.141+0000 7fbb85d9b700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbb7003c9a0 0x7fbb7003ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.142+0000 7fbb85d9b700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbb7003c9a0 0x7fbb7003ed90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c003a10 tx=0x7fbb7c0092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.142+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fbb7800fec0 con 0x7fbb7003c9a0 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.149+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/3667275900 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fbb88068f20 con 0x7fbb7003c9a0 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.149+0000 7fbb777fe700 1 -- 172.21.15.38:0/3667275900 <== mgr.14152 v2:172.21.15.38:6800/2716021457 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fbb88068f20 con 0x7fbb7003c9a0 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.150+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbb7003c9a0 msgr2=0x7fbb7003ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:27.832 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.150+0000 7fbb8cbd6700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbb7003c9a0 0x7fbb7003ed90 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fbb7c003a10 tx=0x7fbb7c0092b0 comp rx=0 tx=0).stop 2024-03-26T06:28:27.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.150+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 msgr2=0x7fbb8819b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:27.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.150+0000 7fbb8cbd6700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8819b8e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbb780116a0 tx=0x7fbb78011ba0 comp rx=0 tx=0).stop 2024-03-26T06:28:27.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.150+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/3667275900 shutdown_connections 2024-03-26T06:28:27.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.150+0000 7fbb8cbd6700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbb7003c9a0 0x7fbb7003ed90 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:27.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.150+0000 7fbb8cbd6700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb88108420 0x7fbb8819b8e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:27.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.150+0000 7fbb8cbd6700 1 --2- 172.21.15.38:0/3667275900 >> [v2:172.21.15.38:6800/4246298230,v1:172.21.15.38:6801/4246298230] conn(0x7fbb70038f20 0x7fbb7003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:27.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.150+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/3667275900 >> 172.21.15.38:0/3667275900 conn(0x7fbb88101e10 msgr2=0x7fbb88102970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:27.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.150+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/3667275900 shutdown_connections 2024-03-26T06:28:27.833 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:27.150+0000 7fbb8cbd6700 1 -- 172.21.15.38:0/3667275900 wait complete. 2024-03-26T06:28:27.833 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 9 is available 2024-03-26T06:28:27.833 INFO:teuthology.orchestra.run.smithi038.stdout:Generating a dashboard self-signed certificate... 2024-03-26T06:28:28.511 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:28 smithi038 ceph-mon[32265]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T06:28:28.511 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:28 smithi038 ceph-mon[32265]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T06:28:29.310 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:29 smithi038 ceph-mon[32265]: mgrmap e12: x(active, since 2s) 2024-03-26T06:28:29.310 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:29 smithi038 ceph-mon[32265]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:29.310 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:29.311 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:29.311 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:29.318 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T06:28:29.318 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.291+0000 7f52848fb700 1 Processor -- start 2024-03-26T06:28:29.318 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.291+0000 7f52848fb700 1 -- start start 2024-03-26T06:28:29.318 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.291+0000 7f52848fb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f5280107db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.291+0000 7f52848fb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5280108380 con 0x7f5280107990 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.292+0000 7f527e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f5280107db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.292+0000 7f527e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f5280107db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41748/0 (socket says 172.21.15.38:41748) 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.292+0000 7f527e59c700 1 -- 172.21.15.38:0/4141716354 learned_addr learned my addr 172.21.15.38:0/4141716354 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.292+0000 7f527e59c700 1 -- 172.21.15.38:0/4141716354 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5280108b90 con 0x7f5280107990 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.292+0000 7f527e59c700 1 --2- 172.21.15.38:0/4141716354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f5280107db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5278009e80 tx=0x7f527800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb974425819ca722 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.292+0000 7f527d59a700 1 -- 172.21.15.38:0/4141716354 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5278017070 con 0x7f5280107990 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.292+0000 7f527d59a700 1 -- 172.21.15.38:0/4141716354 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f527800b8c0 con 0x7f5280107990 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.292+0000 7f527d59a700 1 -- 172.21.15.38:0/4141716354 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f52780045c0 con 0x7f5280107990 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.293+0000 7f52848fb700 1 -- 172.21.15.38:0/4141716354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 msgr2=0x7f5280107db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.293+0000 7f52848fb700 1 --2- 172.21.15.38:0/4141716354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f5280107db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5278009e80 tx=0x7f527800b330 comp rx=0 tx=0).stop 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.293+0000 7f52848fb700 1 -- 172.21.15.38:0/4141716354 shutdown_connections 2024-03-26T06:28:29.319 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.293+0000 7f52848fb700 1 --2- 172.21.15.38:0/4141716354 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f5280107db0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.293+0000 7f52848fb700 1 -- 172.21.15.38:0/4141716354 >> 172.21.15.38:0/4141716354 conn(0x7f5280102f70 msgr2=0x7f5280105390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.293+0000 7f52848fb700 1 -- 172.21.15.38:0/4141716354 shutdown_connections 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.293+0000 7f52848fb700 1 -- 172.21.15.38:0/4141716354 wait complete. 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.294+0000 7f52848fb700 1 Processor -- start 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.294+0000 7f52848fb700 1 -- start start 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.294+0000 7f52848fb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f528007daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.294+0000 7f52848fb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f528010bf40 con 0x7f5280107990 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.294+0000 7f527e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f528007daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.294+0000 7f527e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f528007daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41752/0 (socket says 172.21.15.38:41752) 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.294+0000 7f527e59c700 1 -- 172.21.15.38:0/3277347355 learned_addr learned my addr 172.21.15.38:0/3277347355 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.294+0000 7f527e59c700 1 -- 172.21.15.38:0/3277347355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528007e030 con 0x7f5280107990 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.295+0000 7f527e59c700 1 --2- 172.21.15.38:0/3277347355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f528007daf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5278003fa0 tx=0x7f5278004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.295+0000 7f52777fe700 1 -- 172.21.15.38:0/3277347355 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5278020020 con 0x7f5280107990 2024-03-26T06:28:29.320 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.295+0000 7f52777fe700 1 -- 172.21.15.38:0/3277347355 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f527800bba0 con 0x7f5280107990 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.295+0000 7f52777fe700 1 -- 172.21.15.38:0/3277347355 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5278017040 con 0x7f5280107990 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.295+0000 7f52848fb700 1 -- 172.21.15.38:0/3277347355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528007c220 con 0x7f5280107990 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.295+0000 7f52848fb700 1 -- 172.21.15.38:0/3277347355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528007c760 con 0x7f5280107990 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.296+0000 7f52777fe700 1 -- 172.21.15.38:0/3277347355 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f5278023020 con 0x7f5280107990 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.296+0000 7f52848fb700 1 -- 172.21.15.38:0/3277347355 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5280068f20 con 0x7f5280107990 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.296+0000 7f52777fe700 1 --2- 172.21.15.38:0/3277347355 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f526c038f80 0x7f526c03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.296+0000 7f52777fe700 1 -- 172.21.15.38:0/3277347355 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f527800f050 con 0x7f5280107990 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.296+0000 7f527dd9b700 1 --2- 172.21.15.38:0/3277347355 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f526c038f80 0x7f526c03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.296+0000 7f527dd9b700 1 --2- 172.21.15.38:0/3277347355 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f526c038f80 0x7f526c03b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5268006fd0 tx=0x7f5268006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.301+0000 7f52777fe700 1 -- 172.21.15.38:0/3277347355 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f527802a020 con 0x7f5280107990 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.509+0000 7f52848fb700 1 -- 172.21.15.38:0/3277347355 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f528007cf40 con 0x7f526c038f80 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.689+0000 7f52777fe700 1 -- 172.21.15.38:0/3277347355 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f528007cf40 con 0x7f526c038f80 2024-03-26T06:28:29.321 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.691+0000 7f52848fb700 1 -- 172.21.15.38:0/3277347355 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f526c038f80 msgr2=0x7f526c03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:29.322 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.691+0000 7f52848fb700 1 --2- 172.21.15.38:0/3277347355 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f526c038f80 0x7f526c03b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5268006fd0 tx=0x7f5268006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:29.322 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.691+0000 7f52848fb700 1 -- 172.21.15.38:0/3277347355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 msgr2=0x7f528007daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:29.322 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.691+0000 7f52848fb700 1 --2- 172.21.15.38:0/3277347355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f528007daf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5278003fa0 tx=0x7f5278004080 comp rx=0 tx=0).stop 2024-03-26T06:28:29.322 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.691+0000 7f52848fb700 1 -- 172.21.15.38:0/3277347355 shutdown_connections 2024-03-26T06:28:29.322 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.691+0000 7f52848fb700 1 --2- 172.21.15.38:0/3277347355 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f526c038f80 0x7f526c03b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:29.322 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.691+0000 7f52848fb700 1 --2- 172.21.15.38:0/3277347355 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5280107990 0x7f528007daf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:29.322 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.691+0000 7f52848fb700 1 -- 172.21.15.38:0/3277347355 >> 172.21.15.38:0/3277347355 conn(0x7f5280102f70 msgr2=0x7f5280103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:29.322 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.691+0000 7f52848fb700 1 -- 172.21.15.38:0/3277347355 shutdown_connections 2024-03-26T06:28:29.322 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:28.691+0000 7f52848fb700 1 -- 172.21.15.38:0/3277347355 wait complete. 2024-03-26T06:28:29.322 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial admin user... 2024-03-26T06:28:30.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ka28EN8pI2fHkd5qHmI6yOXEcg0NXP.tVwOgj1dbvP8Uh6woI22nK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711434510, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T06:28:30.802 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.734+0000 7f57504d6700 1 Processor -- start 2024-03-26T06:28:30.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.734+0000 7f57504d6700 1 -- start start 2024-03-26T06:28:30.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.734+0000 7f57504d6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574810a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:30.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.734+0000 7f57504d6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574807bd70 con 0x7f5748108490 2024-03-26T06:28:30.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.735+0000 7f574e272700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574810a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:30.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.735+0000 7f574e272700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574810a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41758/0 (socket says 172.21.15.38:41758) 2024-03-26T06:28:30.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.735+0000 7f574e272700 1 -- 172.21.15.38:0/3645628029 learned_addr learned my addr 172.21.15.38:0/3645628029 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:30.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.735+0000 7f574e272700 1 -- 172.21.15.38:0/3645628029 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574810adc0 con 0x7f5748108490 2024-03-26T06:28:30.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.735+0000 7f574e272700 1 --2- 172.21.15.38:0/3645628029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574810a880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57380098a0 tx=0x7f5738009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ee022b89e9122fd server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:30.803 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.735+0000 7f574d270700 1 -- 172.21.15.38:0/3645628029 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5738004160 con 0x7f5748108490 2024-03-26T06:28:30.804 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.735+0000 7f574d270700 1 -- 172.21.15.38:0/3645628029 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5738004300 con 0x7f5748108490 2024-03-26T06:28:30.804 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.735+0000 7f574d270700 1 -- 172.21.15.38:0/3645628029 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5738011b20 con 0x7f5748108490 2024-03-26T06:28:30.804 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.736+0000 7f57504d6700 1 -- 172.21.15.38:0/3645628029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 msgr2=0x7f574810a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:30.804 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.736+0000 7f57504d6700 1 --2- 172.21.15.38:0/3645628029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574810a880 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f57380098a0 tx=0x7f5738009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:30.804 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.736+0000 7f57504d6700 1 -- 172.21.15.38:0/3645628029 shutdown_connections 2024-03-26T06:28:30.804 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.736+0000 7f57504d6700 1 --2- 172.21.15.38:0/3645628029 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574810a880 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:30.804 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.736+0000 7f57504d6700 1 -- 172.21.15.38:0/3645628029 >> 172.21.15.38:0/3645628029 conn(0x7f5748101e40 msgr2=0x7f5748104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:30.804 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.736+0000 7f57504d6700 1 -- 172.21.15.38:0/3645628029 shutdown_connections 2024-03-26T06:28:30.804 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.736+0000 7f57504d6700 1 -- 172.21.15.38:0/3645628029 wait complete. 2024-03-26T06:28:30.805 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f57504d6700 1 Processor -- start 2024-03-26T06:28:30.805 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f57504d6700 1 -- start start 2024-03-26T06:28:30.805 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f57504d6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574819b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:30.805 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f57504d6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574810bd70 con 0x7f5748108490 2024-03-26T06:28:30.805 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f574e272700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574819b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:30.805 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f574e272700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574819b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41762/0 (socket says 172.21.15.38:41762) 2024-03-26T06:28:30.805 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f574e272700 1 -- 172.21.15.38:0/2505150663 learned_addr learned my addr 172.21.15.38:0/2505150663 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:30.806 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f574e272700 1 -- 172.21.15.38:0/2505150663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574819be10 con 0x7f5748108490 2024-03-26T06:28:30.806 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f574e272700 1 --2- 172.21.15.38:0/2505150663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574819b8d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f57380116a0 tx=0x7f5738011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:30.806 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f573f7fe700 1 -- 172.21.15.38:0/2505150663 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5738003a40 con 0x7f5748108490 2024-03-26T06:28:30.806 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f573f7fe700 1 -- 172.21.15.38:0/2505150663 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5738003be0 con 0x7f5748108490 2024-03-26T06:28:30.806 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f57504d6700 1 -- 172.21.15.38:0/2505150663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574819c0a0 con 0x7f5748108490 2024-03-26T06:28:30.806 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.737+0000 7f573f7fe700 1 -- 172.21.15.38:0/2505150663 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f573801a750 con 0x7f5748108490 2024-03-26T06:28:30.806 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.738+0000 7f57504d6700 1 -- 172.21.15.38:0/2505150663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574819c500 con 0x7f5748108490 2024-03-26T06:28:30.806 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.738+0000 7f573f7fe700 1 -- 172.21.15.38:0/2505150663 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f573801aa50 con 0x7f5748108490 2024-03-26T06:28:30.807 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.739+0000 7f573f7fe700 1 --2- 172.21.15.38:0/2505150663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5734038f20 0x7f573403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:30.807 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.739+0000 7f57504d6700 1 -- 172.21.15.38:0/2505150663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5748068f20 con 0x7f5748108490 2024-03-26T06:28:30.807 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.739+0000 7f573f7fe700 1 -- 172.21.15.38:0/2505150663 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5738024aa0 con 0x7f5748108490 2024-03-26T06:28:30.807 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.739+0000 7f574da71700 1 --2- 172.21.15.38:0/2505150663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5734038f20 0x7f573403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:30.807 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.739+0000 7f574da71700 1 --2- 172.21.15.38:0/2505150663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5734038f20 0x7f573403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5744006fd0 tx=0x7f5744006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:30.807 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.744+0000 7f573f7fe700 1 -- 172.21.15.38:0/2505150663 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5738011e20 con 0x7f5748108490 2024-03-26T06:28:30.807 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:29.972+0000 7f57504d6700 1 -- 172.21.15.38:0/2505150663 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f574819ef30 con 0x7f5734038f20 2024-03-26T06:28:30.807 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:30.220+0000 7f573f7fe700 1 -- 172.21.15.38:0/2505150663 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f574819ef30 con 0x7f5734038f20 2024-03-26T06:28:30.808 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:30.223+0000 7f57504d6700 1 -- 172.21.15.38:0/2505150663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5734038f20 msgr2=0x7f573403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:30.808 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:30.223+0000 7f57504d6700 1 --2- 172.21.15.38:0/2505150663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5734038f20 0x7f573403b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5744006fd0 tx=0x7f5744006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:30.808 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:30.223+0000 7f57504d6700 1 -- 172.21.15.38:0/2505150663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 msgr2=0x7f574819b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:30.808 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:30.223+0000 7f57504d6700 1 --2- 172.21.15.38:0/2505150663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574819b8d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f57380116a0 tx=0x7f5738011ba0 comp rx=0 tx=0).stop 2024-03-26T06:28:30.808 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:30.223+0000 7f57504d6700 1 -- 172.21.15.38:0/2505150663 shutdown_connections 2024-03-26T06:28:30.808 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:30.223+0000 7f57504d6700 1 --2- 172.21.15.38:0/2505150663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5734038f20 0x7f573403b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:30.808 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:30.223+0000 7f57504d6700 1 --2- 172.21.15.38:0/2505150663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5748108490 0x7f574819b8d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:30.808 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:30.223+0000 7f57504d6700 1 -- 172.21.15.38:0/2505150663 >> 172.21.15.38:0/2505150663 conn(0x7f5748101e40 msgr2=0x7f5748102930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:30.809 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:30.223+0000 7f57504d6700 1 -- 172.21.15.38:0/2505150663 shutdown_connections 2024-03-26T06:28:30.809 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:30.223+0000 7f57504d6700 1 -- 172.21.15.38:0/2505150663 wait complete. 2024-03-26T06:28:30.809 INFO:teuthology.orchestra.run.smithi038.stdout:Fetching dashboard port number... 2024-03-26T06:28:31.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:30 smithi038 ceph-mon[32265]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:31.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:30 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:31.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:30 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:32.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:32 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2621063189' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T06:28:32.630 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T06:28:32.630 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.176+0000 7f2b574b2700 1 Processor -- start 2024-03-26T06:28:32.630 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.176+0000 7f2b574b2700 1 -- start start 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.176+0000 7f2b574b2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b50107e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.176+0000 7f2b574b2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b50108400 con 0x7f2b50107a10 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.176+0000 7f2b5524e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b50107e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.176+0000 7f2b5524e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b50107e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41764/0 (socket says 172.21.15.38:41764) 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.176+0000 7f2b5524e700 1 -- 172.21.15.38:0/4056788075 learned_addr learned my addr 172.21.15.38:0/4056788075 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.176+0000 7f2b5524e700 1 -- 172.21.15.38:0/4056788075 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b50108b70 con 0x7f2b50107a10 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.176+0000 7f2b5524e700 1 --2- 172.21.15.38:0/4056788075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b50107e30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0098a0 tx=0x7f2b4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729ff7da7e14327b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.177+0000 7f2b47fff700 1 -- 172.21.15.38:0/4056788075 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b4c004160 con 0x7f2b50107a10 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.177+0000 7f2b47fff700 1 -- 172.21.15.38:0/4056788075 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b4c004300 con 0x7f2b50107a10 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.177+0000 7f2b47fff700 1 -- 172.21.15.38:0/4056788075 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b4c011b20 con 0x7f2b50107a10 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.177+0000 7f2b574b2700 1 -- 172.21.15.38:0/4056788075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 msgr2=0x7f2b50107e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.177+0000 7f2b574b2700 1 --2- 172.21.15.38:0/4056788075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b50107e30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0098a0 tx=0x7f2b4c009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:32.631 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.177+0000 7f2b574b2700 1 -- 172.21.15.38:0/4056788075 shutdown_connections 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.177+0000 7f2b574b2700 1 --2- 172.21.15.38:0/4056788075 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b50107e30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.177+0000 7f2b574b2700 1 -- 172.21.15.38:0/4056788075 >> 172.21.15.38:0/4056788075 conn(0x7f2b50102fb0 msgr2=0x7f2b501053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.177+0000 7f2b574b2700 1 -- 172.21.15.38:0/4056788075 shutdown_connections 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.177+0000 7f2b574b2700 1 -- 172.21.15.38:0/4056788075 wait complete. 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.178+0000 7f2b574b2700 1 Processor -- start 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.178+0000 7f2b574b2700 1 -- start start 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.178+0000 7f2b574b2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b5019b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.178+0000 7f2b574b2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5010bf20 con 0x7f2b50107a10 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.178+0000 7f2b5524e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b5019b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.178+0000 7f2b5524e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b5019b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41768/0 (socket says 172.21.15.38:41768) 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.178+0000 7f2b5524e700 1 -- 172.21.15.38:0/2621063189 learned_addr learned my addr 172.21.15.38:0/2621063189 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.178+0000 7f2b5524e700 1 -- 172.21.15.38:0/2621063189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5019be20 con 0x7f2b50107a10 2024-03-26T06:28:32.632 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.179+0000 7f2b5524e700 1 --2- 172.21.15.38:0/2621063189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b5019b8e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0117e0 tx=0x7f2b4c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.179+0000 7f2b467fc700 1 -- 172.21.15.38:0/2621063189 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b4c003bf0 con 0x7f2b50107a10 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.179+0000 7f2b574b2700 1 -- 172.21.15.38:0/2621063189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5019c0b0 con 0x7f2b50107a10 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.179+0000 7f2b467fc700 1 -- 172.21.15.38:0/2621063189 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2b4c01a910 con 0x7f2b50107a10 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.179+0000 7f2b467fc700 1 -- 172.21.15.38:0/2621063189 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2b4c024970 con 0x7f2b50107a10 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.179+0000 7f2b574b2700 1 -- 172.21.15.38:0/2621063189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5019c510 con 0x7f2b50107a10 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.180+0000 7f2b467fc700 1 -- 172.21.15.38:0/2621063189 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f2b4c022020 con 0x7f2b50107a10 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.180+0000 7f2b574b2700 1 -- 172.21.15.38:0/2621063189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b50063410 con 0x7f2b50107a10 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.180+0000 7f2b467fc700 1 --2- 172.21.15.38:0/2621063189 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2b3c038f70 0x7f2b3c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.180+0000 7f2b467fc700 1 -- 172.21.15.38:0/2621063189 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2b4c04d8d0 con 0x7f2b50107a10 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.180+0000 7f2b54a4d700 1 --2- 172.21.15.38:0/2621063189 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2b3c038f70 0x7f2b3c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.181+0000 7f2b54a4d700 1 --2- 172.21.15.38:0/2621063189 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2b3c038f70 0x7f2b3c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2b40006fd0 tx=0x7f2b40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.185+0000 7f2b467fc700 1 -- 172.21.15.38:0/2621063189 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2b4c02cdf0 con 0x7f2b50107a10 2024-03-26T06:28:32.633 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.397+0000 7f2b574b2700 1 -- 172.21.15.38:0/2621063189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f2b5019eee0 con 0x7f2b50107a10 2024-03-26T06:28:32.634 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.398+0000 7f2b467fc700 1 -- 172.21.15.38:0/2621063189 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f2b4c00b5a0 con 0x7f2b50107a10 2024-03-26T06:28:32.634 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.400+0000 7f2b574b2700 1 -- 172.21.15.38:0/2621063189 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2b3c038f70 msgr2=0x7f2b3c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:32.634 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.400+0000 7f2b574b2700 1 --2- 172.21.15.38:0/2621063189 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2b3c038f70 0x7f2b3c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2b40006fd0 tx=0x7f2b40006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:32.634 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.400+0000 7f2b574b2700 1 -- 172.21.15.38:0/2621063189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 msgr2=0x7f2b5019b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:32.634 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.400+0000 7f2b574b2700 1 --2- 172.21.15.38:0/2621063189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b5019b8e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c0117e0 tx=0x7f2b4c011ba0 comp rx=0 tx=0).stop 2024-03-26T06:28:32.634 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.400+0000 7f2b574b2700 1 -- 172.21.15.38:0/2621063189 shutdown_connections 2024-03-26T06:28:32.634 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.400+0000 7f2b574b2700 1 --2- 172.21.15.38:0/2621063189 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2b3c038f70 0x7f2b3c03b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:32.634 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.400+0000 7f2b574b2700 1 --2- 172.21.15.38:0/2621063189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2b50107a10 0x7f2b5019b8e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:32.634 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.400+0000 7f2b574b2700 1 -- 172.21.15.38:0/2621063189 >> 172.21.15.38:0/2621063189 conn(0x7f2b50102fb0 msgr2=0x7f2b50103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:32.634 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.400+0000 7f2b574b2700 1 -- 172.21.15.38:0/2621063189 shutdown_connections 2024-03-26T06:28:32.634 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:31.400+0000 7f2b574b2700 1 -- 172.21.15.38:0/2621063189 wait complete. 2024-03-26T06:28:32.640 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T06:28:32.640 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2024-03-26T06:28:32.648 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T06:28:32.648 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2024-03-26T06:28:32.648 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2024-03-26T06:28:32.648 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T06:28:32.650 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph Dashboard is now available at: 2024-03-26T06:28:32.650 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:28:32.650 INFO:teuthology.orchestra.run.smithi038.stdout: URL: https://smithi038.front.sepia.ceph.com:8443/ 2024-03-26T06:28:32.650 INFO:teuthology.orchestra.run.smithi038.stdout: User: admin 2024-03-26T06:28:32.650 INFO:teuthology.orchestra.run.smithi038.stdout: Password: 6xxzcinfv9 2024-03-26T06:28:32.651 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:28:32.651 INFO:teuthology.orchestra.run.smithi038.stdout:Saving cluster configuration to /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config directory 2024-03-26T06:28:32.651 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling autotune for osd_memory_target 2024-03-26T06:28:34.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:34 smithi038 ceph-mon[32265]: mgrmap e13: x(active, since 6s) 2024-03-26T06:28:34.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.472+0000 7f4372f96700 1 Processor -- start 2024-03-26T06:28:34.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.472+0000 7f4372f96700 1 -- start start 2024-03-26T06:28:34.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.472+0000 7f4372f96700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c106ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:34.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.472+0000 7f4372f96700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c107270 con 0x7f436c106880 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.472+0000 7f4370d32700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c106ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.472+0000 7f4370d32700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c106ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41778/0 (socket says 172.21.15.38:41778) 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.472+0000 7f4370d32700 1 -- 172.21.15.38:0/3631644440 learned_addr learned my addr 172.21.15.38:0/3631644440 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.473+0000 7f4370d32700 1 -- 172.21.15.38:0/3631644440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c107a80 con 0x7f436c106880 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.473+0000 7f4370d32700 1 --2- 172.21.15.38:0/3631644440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c106ca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f43600098a0 tx=0x7f4360009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=581e55d87eb3fb11 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.473+0000 7f436b7fe700 1 -- 172.21.15.38:0/3631644440 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4360004160 con 0x7f436c106880 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.473+0000 7f436b7fe700 1 -- 172.21.15.38:0/3631644440 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4360004300 con 0x7f436c106880 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.473+0000 7f436b7fe700 1 -- 172.21.15.38:0/3631644440 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4360011b20 con 0x7f436c106880 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.474+0000 7f4372f96700 1 -- 172.21.15.38:0/3631644440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 msgr2=0x7f436c106ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.474+0000 7f4372f96700 1 --2- 172.21.15.38:0/3631644440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c106ca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f43600098a0 tx=0x7f4360009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.474+0000 7f4372f96700 1 -- 172.21.15.38:0/3631644440 shutdown_connections 2024-03-26T06:28:34.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.474+0000 7f4372f96700 1 --2- 172.21.15.38:0/3631644440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c106ca0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.474+0000 7f4372f96700 1 -- 172.21.15.38:0/3631644440 >> 172.21.15.38:0/3631644440 conn(0x7f436c101e80 msgr2=0x7f436c1042a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.474+0000 7f4372f96700 1 -- 172.21.15.38:0/3631644440 shutdown_connections 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.474+0000 7f4372f96700 1 -- 172.21.15.38:0/3631644440 wait complete. 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.474+0000 7f4372f96700 1 Processor -- start 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.474+0000 7f4372f96700 1 -- start start 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.475+0000 7f4372f96700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c19b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.475+0000 7f4372f96700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c10ae30 con 0x7f436c106880 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.475+0000 7f4370d32700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c19b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.475+0000 7f4370d32700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c19b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41794/0 (socket says 172.21.15.38:41794) 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.475+0000 7f4370d32700 1 -- 172.21.15.38:0/298840475 learned_addr learned my addr 172.21.15.38:0/298840475 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.475+0000 7f4370d32700 1 -- 172.21.15.38:0/298840475 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c19be80 con 0x7f436c106880 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.475+0000 7f4370d32700 1 --2- 172.21.15.38:0/298840475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c19b940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43600117e0 tx=0x7f4360011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:34.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.475+0000 7f4369ffb700 1 -- 172.21.15.38:0/298840475 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4360003bf0 con 0x7f436c106880 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.475+0000 7f4369ffb700 1 -- 172.21.15.38:0/298840475 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f436001a910 con 0x7f436c106880 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.475+0000 7f4372f96700 1 -- 172.21.15.38:0/298840475 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436c19c110 con 0x7f436c106880 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.475+0000 7f4369ffb700 1 -- 172.21.15.38:0/298840475 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f43600249a0 con 0x7f436c106880 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.476+0000 7f4372f96700 1 -- 172.21.15.38:0/298840475 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436c19c570 con 0x7f436c106880 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.476+0000 7f4369ffb700 1 -- 172.21.15.38:0/298840475 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f4360022020 con 0x7f436c106880 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.477+0000 7f4372f96700 1 -- 172.21.15.38:0/298840475 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436c063410 con 0x7f436c106880 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.477+0000 7f4369ffb700 1 --2- 172.21.15.38:0/298840475 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f435c039020 0x7f435c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.477+0000 7f4369ffb700 1 -- 172.21.15.38:0/298840475 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f436004d360 con 0x7f436c106880 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.477+0000 7f436bfff700 1 --2- 172.21.15.38:0/298840475 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f435c039020 0x7f435c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.477+0000 7f436bfff700 1 --2- 172.21.15.38:0/298840475 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f435c039020 0x7f435c03b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4358006fd0 tx=0x7f4358006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.482+0000 7f4369ffb700 1 -- 172.21.15.38:0/298840475 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4360011e20 con 0x7f436c106880 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.685+0000 7f4372f96700 1 -- 172.21.15.38:0/298840475 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f436c0629f0 con 0x7f436c106880 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.685+0000 7f4369ffb700 1 -- 172.21.15.38:0/298840475 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f4360004300 con 0x7f436c106880 2024-03-26T06:28:34.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.687+0000 7f4372f96700 1 -- 172.21.15.38:0/298840475 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f435c039020 msgr2=0x7f435c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:34.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.687+0000 7f4372f96700 1 --2- 172.21.15.38:0/298840475 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f435c039020 0x7f435c03b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4358006fd0 tx=0x7f4358006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:34.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.687+0000 7f4372f96700 1 -- 172.21.15.38:0/298840475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 msgr2=0x7f436c19b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:34.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.687+0000 7f4372f96700 1 --2- 172.21.15.38:0/298840475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c19b940 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43600117e0 tx=0x7f4360011c10 comp rx=0 tx=0).stop 2024-03-26T06:28:34.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.687+0000 7f4372f96700 1 -- 172.21.15.38:0/298840475 shutdown_connections 2024-03-26T06:28:34.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.687+0000 7f4372f96700 1 --2- 172.21.15.38:0/298840475 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f435c039020 0x7f435c03b4e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:34.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.687+0000 7f4372f96700 1 --2- 172.21.15.38:0/298840475 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f436c106880 0x7f436c19b940 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:34.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.687+0000 7f4372f96700 1 -- 172.21.15.38:0/298840475 >> 172.21.15.38:0/298840475 conn(0x7f436c101e80 msgr2=0x7f436c102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:34.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.688+0000 7f4372f96700 1 -- 172.21.15.38:0/298840475 shutdown_connections 2024-03-26T06:28:34.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:33.688+0000 7f4372f96700 1 -- 172.21.15.38:0/298840475 wait complete. 2024-03-26T06:28:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.739+0000 7f72aab92700 1 Processor -- start 2024-03-26T06:28:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.739+0000 7f72aab92700 1 -- start start 2024-03-26T06:28:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.739+0000 7f72aab92700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a4107e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:35.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.739+0000 7f72aab92700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a4108400 con 0x7f72a4107a10 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.739+0000 7f72a892e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a4107e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.739+0000 7f72a892e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a4107e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46780/0 (socket says 172.21.15.38:46780) 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.739+0000 7f72a892e700 1 -- 172.21.15.38:0/1622054077 learned_addr learned my addr 172.21.15.38:0/1622054077 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.740+0000 7f72a892e700 1 -- 172.21.15.38:0/1622054077 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4108b70 con 0x7f72a4107a10 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.740+0000 7f72a892e700 1 --2- 172.21.15.38:0/1622054077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a4107e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72980098a0 tx=0x7f7298009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b258a2bce8f34b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.740+0000 7f72a37fe700 1 -- 172.21.15.38:0/1622054077 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7298004160 con 0x7f72a4107a10 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.740+0000 7f72a37fe700 1 -- 172.21.15.38:0/1622054077 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7298004300 con 0x7f72a4107a10 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.740+0000 7f72a37fe700 1 -- 172.21.15.38:0/1622054077 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7298011b20 con 0x7f72a4107a10 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.741+0000 7f72aab92700 1 -- 172.21.15.38:0/1622054077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 msgr2=0x7f72a4107e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.741+0000 7f72aab92700 1 --2- 172.21.15.38:0/1622054077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a4107e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f72980098a0 tx=0x7f7298009bb0 comp rx=0 tx=0).stop 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.741+0000 7f72aab92700 1 -- 172.21.15.38:0/1622054077 shutdown_connections 2024-03-26T06:28:35.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.741+0000 7f72aab92700 1 --2- 172.21.15.38:0/1622054077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a4107e30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.741+0000 7f72aab92700 1 -- 172.21.15.38:0/1622054077 >> 172.21.15.38:0/1622054077 conn(0x7f72a4102fb0 msgr2=0x7f72a41053f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.741+0000 7f72aab92700 1 -- 172.21.15.38:0/1622054077 shutdown_connections 2024-03-26T06:28:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.741+0000 7f72aab92700 1 -- 172.21.15.38:0/1622054077 wait complete. 2024-03-26T06:28:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.741+0000 7f72aab92700 1 Processor -- start 2024-03-26T06:28:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.741+0000 7f72aab92700 1 -- start start 2024-03-26T06:28:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.741+0000 7f72aab92700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a419b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:35.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.741+0000 7f72aab92700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a410bf20 con 0x7f72a4107a10 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.742+0000 7f72a892e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a419b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.742+0000 7f72a892e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a419b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46792/0 (socket says 172.21.15.38:46792) 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.742+0000 7f72a892e700 1 -- 172.21.15.38:0/2986019497 learned_addr learned my addr 172.21.15.38:0/2986019497 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.742+0000 7f72a892e700 1 -- 172.21.15.38:0/2986019497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a419be20 con 0x7f72a4107a10 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.742+0000 7f72a892e700 1 --2- 172.21.15.38:0/2986019497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a419b8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f72980117e0 tx=0x7f7298011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.742+0000 7f72a1ffb700 1 -- 172.21.15.38:0/2986019497 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7298003bf0 con 0x7f72a4107a10 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.742+0000 7f72a1ffb700 1 -- 172.21.15.38:0/2986019497 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f729801a910 con 0x7f72a4107a10 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.742+0000 7f72aab92700 1 -- 172.21.15.38:0/2986019497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a419c0b0 con 0x7f72a4107a10 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.742+0000 7f72a1ffb700 1 -- 172.21.15.38:0/2986019497 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7298024970 con 0x7f72a4107a10 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.742+0000 7f72aab92700 1 -- 172.21.15.38:0/2986019497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a419c510 con 0x7f72a4107a10 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.743+0000 7f72a1ffb700 1 -- 172.21.15.38:0/2986019497 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f7298022020 con 0x7f72a4107a10 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.743+0000 7f72a1ffb700 1 --2- 172.21.15.38:0/2986019497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7294039010 0x7f729403b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:35.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.743+0000 7f72a1ffb700 1 -- 172.21.15.38:0/2986019497 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f729804db50 con 0x7f72a4107a10 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.743+0000 7f72a3fff700 1 --2- 172.21.15.38:0/2986019497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7294039010 0x7f729403b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.743+0000 7f72aab92700 1 -- 172.21.15.38:0/2986019497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a4063410 con 0x7f72a4107a10 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.744+0000 7f72a3fff700 1 --2- 172.21.15.38:0/2986019497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7294039010 0x7f729403b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7290006fd0 tx=0x7f7290006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:34.749+0000 7f72a1ffb700 1 -- 172.21.15.38:0/2986019497 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f729800b3e0 con 0x7f72a4107a10 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.071+0000 7f72aab92700 1 -- 172.21.15.38:0/2986019497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f72a419ef20 con 0x7f72a4107a10 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.073+0000 7f72a1ffb700 1 -- 172.21.15.38:0/2986019497 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (secure 0 0 0) 0x7f729802cda0 con 0x7f72a4107a10 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.075+0000 7f72aab92700 1 -- 172.21.15.38:0/2986019497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7294039010 msgr2=0x7f729403b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.075+0000 7f72aab92700 1 --2- 172.21.15.38:0/2986019497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7294039010 0x7f729403b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7290006fd0 tx=0x7f7290006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.075+0000 7f72aab92700 1 -- 172.21.15.38:0/2986019497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 msgr2=0x7f72a419b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.075+0000 7f72aab92700 1 --2- 172.21.15.38:0/2986019497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a419b8e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f72980117e0 tx=0x7f7298011ba0 comp rx=0 tx=0).stop 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.075+0000 7f72aab92700 1 -- 172.21.15.38:0/2986019497 shutdown_connections 2024-03-26T06:28:35.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.075+0000 7f72aab92700 1 --2- 172.21.15.38:0/2986019497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7294039010 0x7f729403b4d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.075+0000 7f72aab92700 1 --2- 172.21.15.38:0/2986019497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72a4107a10 0x7f72a419b8e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.075+0000 7f72aab92700 1 -- 172.21.15.38:0/2986019497 >> 172.21.15.38:0/2986019497 conn(0x7f72a4102fb0 msgr2=0x7f72a4103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.075+0000 7f72aab92700 1 -- 172.21.15.38:0/2986019497 shutdown_connections 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2024-03-26T06:28:35.076+0000 7f72aab92700 1 -- 172.21.15.38:0/2986019497 wait complete. 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:28:35.660 INFO:teuthology.orchestra.run.smithi038.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T06:28:35.661 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:28:35.661 INFO:teuthology.orchestra.run.smithi038.stdout: ceph telemetry on 2024-03-26T06:28:35.661 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:28:35.661 INFO:teuthology.orchestra.run.smithi038.stdout:For more information see: 2024-03-26T06:28:35.661 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:28:35.661 INFO:teuthology.orchestra.run.smithi038.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T06:28:35.661 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:28:35.661 INFO:teuthology.orchestra.run.smithi038.stdout:Bootstrap complete. 2024-03-26T06:28:35.694 INFO:tasks.cephadm:Fetching config... 2024-03-26T06:28:35.694 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:28:35.694 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T06:28:35.710 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T06:28:35.710 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:28:35.710 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T06:28:35.766 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T06:28:35.766 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:28:35.766 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T06:28:35.834 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T06:28:35.834 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:28:35.834 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T06:28:35.890 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T06:28:35.891 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS8CLzDJbbKJNW6ZfsKSGWqdnZgvFnZTZB2X4o2Pc0FvC7Z80cgLODE8fkK5VVevZlSHJ1jRr6I7grFmSZgeFxTdqpOghjuvjlMgt0OaXicSIg7EVA1P1s0Z1IY7rCnsxBb6eqtStQnN5XcFatGaVmKL0NuYSOEtmmYswpgv1pd5BFBAEFH7Rp+AvBCwjxwvTcEhLxlJ+Gqy+8oJuiif7+EdkqZNN4q38w0HPERU0uMTIojwfmfuMjIUKQshTJWcmMCKJQMdADf+yRDn8X/JkzEkZq/2qK04DkktuceE61+iA2K3KzMyZXxbJVwJ2rsidaSwJHHOvbhHOgWVDJtqa2T2wSyNJVHz63hym8Lq15XVi6j/62Y2JlwB4UIYb1lFwkoFrH1N3hU2/S/d18yzUkhzD5J6DXPEHCZKWYf3X944Y6sJbZVn8ueYt5UFoYJ9K/iSEf4xX5JWsACSWttULlnZj4Q9bRjuCgsJ90/slFAz9Hykj0tP4TVBUeeA7AxT0= ceph-c2824556-eb39-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T06:28:35.969 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS8CLzDJbbKJNW6ZfsKSGWqdnZgvFnZTZB2X4o2Pc0FvC7Z80cgLODE8fkK5VVevZlSHJ1jRr6I7grFmSZgeFxTdqpOghjuvjlMgt0OaXicSIg7EVA1P1s0Z1IY7rCnsxBb6eqtStQnN5XcFatGaVmKL0NuYSOEtmmYswpgv1pd5BFBAEFH7Rp+AvBCwjxwvTcEhLxlJ+Gqy+8oJuiif7+EdkqZNN4q38w0HPERU0uMTIojwfmfuMjIUKQshTJWcmMCKJQMdADf+yRDn8X/JkzEkZq/2qK04DkktuceE61+iA2K3KzMyZXxbJVwJ2rsidaSwJHHOvbhHOgWVDJtqa2T2wSyNJVHz63hym8Lq15XVi6j/62Y2JlwB4UIYb1lFwkoFrH1N3hU2/S/d18yzUkhzD5J6DXPEHCZKWYf3X944Y6sJbZVn8ueYt5UFoYJ9K/iSEf4xX5JWsACSWttULlnZj4Q9bRjuCgsJ90/slFAz9Hykj0tP4TVBUeeA7AxT0= ceph-c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:28:35.983 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS8CLzDJbbKJNW6ZfsKSGWqdnZgvFnZTZB2X4o2Pc0FvC7Z80cgLODE8fkK5VVevZlSHJ1jRr6I7grFmSZgeFxTdqpOghjuvjlMgt0OaXicSIg7EVA1P1s0Z1IY7rCnsxBb6eqtStQnN5XcFatGaVmKL0NuYSOEtmmYswpgv1pd5BFBAEFH7Rp+AvBCwjxwvTcEhLxlJ+Gqy+8oJuiif7+EdkqZNN4q38w0HPERU0uMTIojwfmfuMjIUKQshTJWcmMCKJQMdADf+yRDn8X/JkzEkZq/2qK04DkktuceE61+iA2K3KzMyZXxbJVwJ2rsidaSwJHHOvbhHOgWVDJtqa2T2wSyNJVHz63hym8Lq15XVi6j/62Y2JlwB4UIYb1lFwkoFrH1N3hU2/S/d18yzUkhzD5J6DXPEHCZKWYf3X944Y6sJbZVn8ueYt5UFoYJ9K/iSEf4xX5JWsACSWttULlnZj4Q9bRjuCgsJ90/slFAz9Hykj0tP4TVBUeeA7AxT0= ceph-c2824556-eb39-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T06:28:36.028 INFO:teuthology.orchestra.run.smithi121.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS8CLzDJbbKJNW6ZfsKSGWqdnZgvFnZTZB2X4o2Pc0FvC7Z80cgLODE8fkK5VVevZlSHJ1jRr6I7grFmSZgeFxTdqpOghjuvjlMgt0OaXicSIg7EVA1P1s0Z1IY7rCnsxBb6eqtStQnN5XcFatGaVmKL0NuYSOEtmmYswpgv1pd5BFBAEFH7Rp+AvBCwjxwvTcEhLxlJ+Gqy+8oJuiif7+EdkqZNN4q38w0HPERU0uMTIojwfmfuMjIUKQshTJWcmMCKJQMdADf+yRDn8X/JkzEkZq/2qK04DkktuceE61+iA2K3KzMyZXxbJVwJ2rsidaSwJHHOvbhHOgWVDJtqa2T2wSyNJVHz63hym8Lq15XVi6j/62Y2JlwB4UIYb1lFwkoFrH1N3hU2/S/d18yzUkhzD5J6DXPEHCZKWYf3X944Y6sJbZVn8ueYt5UFoYJ9K/iSEf4xX5JWsACSWttULlnZj4Q9bRjuCgsJ90/slFAz9Hykj0tP4TVBUeeA7AxT0= ceph-c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:28:36.044 DEBUG:teuthology.orchestra.run.smithi170:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS8CLzDJbbKJNW6ZfsKSGWqdnZgvFnZTZB2X4o2Pc0FvC7Z80cgLODE8fkK5VVevZlSHJ1jRr6I7grFmSZgeFxTdqpOghjuvjlMgt0OaXicSIg7EVA1P1s0Z1IY7rCnsxBb6eqtStQnN5XcFatGaVmKL0NuYSOEtmmYswpgv1pd5BFBAEFH7Rp+AvBCwjxwvTcEhLxlJ+Gqy+8oJuiif7+EdkqZNN4q38w0HPERU0uMTIojwfmfuMjIUKQshTJWcmMCKJQMdADf+yRDn8X/JkzEkZq/2qK04DkktuceE61+iA2K3KzMyZXxbJVwJ2rsidaSwJHHOvbhHOgWVDJtqa2T2wSyNJVHz63hym8Lq15XVi6j/62Y2JlwB4UIYb1lFwkoFrH1N3hU2/S/d18yzUkhzD5J6DXPEHCZKWYf3X944Y6sJbZVn8ueYt5UFoYJ9K/iSEf4xX5JWsACSWttULlnZj4Q9bRjuCgsJ90/slFAz9Hykj0tP4TVBUeeA7AxT0= ceph-c2824556-eb39-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T06:28:36.090 INFO:teuthology.orchestra.run.smithi170.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDS8CLzDJbbKJNW6ZfsKSGWqdnZgvFnZTZB2X4o2Pc0FvC7Z80cgLODE8fkK5VVevZlSHJ1jRr6I7grFmSZgeFxTdqpOghjuvjlMgt0OaXicSIg7EVA1P1s0Z1IY7rCnsxBb6eqtStQnN5XcFatGaVmKL0NuYSOEtmmYswpgv1pd5BFBAEFH7Rp+AvBCwjxwvTcEhLxlJ+Gqy+8oJuiif7+EdkqZNN4q38w0HPERU0uMTIojwfmfuMjIUKQshTJWcmMCKJQMdADf+yRDn8X/JkzEkZq/2qK04DkktuceE61+iA2K3KzMyZXxbJVwJ2rsidaSwJHHOvbhHOgWVDJtqa2T2wSyNJVHz63hym8Lq15XVi6j/62Y2JlwB4UIYb1lFwkoFrH1N3hU2/S/d18yzUkhzD5J6DXPEHCZKWYf3X944Y6sJbZVn8ueYt5UFoYJ9K/iSEf4xX5JWsACSWttULlnZj4Q9bRjuCgsJ90/slFAz9Hykj0tP4TVBUeeA7AxT0= ceph-c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:28:36.105 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T06:28:36.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:36 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2986019497' entity='client.admin' 2024-03-26T06:28:36.352 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:28:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.528+0000 7f05ac66b700 1 -- 172.21.15.38:0/3028666055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a41022a0 msgr2=0x7f05a4104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.528+0000 7f05ac66b700 1 --2- 172.21.15.38:0/3028666055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a41022a0 0x7f05a4104690 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0594009910 tx=0x7f0594009c20 comp rx=0 tx=0).stop 2024-03-26T06:28:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.528+0000 7f05ac66b700 1 -- 172.21.15.38:0/3028666055 shutdown_connections 2024-03-26T06:28:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.528+0000 7f05ac66b700 1 --2- 172.21.15.38:0/3028666055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a41022a0 0x7f05a4104690 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.528+0000 7f05ac66b700 1 -- 172.21.15.38:0/3028666055 >> 172.21.15.38:0/3028666055 conn(0x7f05a40fbcb0 msgr2=0x7f05a40fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.528+0000 7f05ac66b700 1 -- 172.21.15.38:0/3028666055 shutdown_connections 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.528+0000 7f05ac66b700 1 -- 172.21.15.38:0/3028666055 wait complete. 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.529+0000 7f05ac66b700 1 Processor -- start 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.529+0000 7f05ac66b700 1 -- start start 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.529+0000 7f05ac66b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a41022a0 0x7f05a4196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.529+0000 7f05ac66b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a4107840 con 0x7f05a41022a0 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.529+0000 7f05aa407700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a41022a0 0x7f05a4196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.529+0000 7f05aa407700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a41022a0 0x7f05a4196e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46816/0 (socket says 172.21.15.38:46816) 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.529+0000 7f05aa407700 1 -- 172.21.15.38:0/91524174 learned_addr learned my addr 172.21.15.38:0/91524174 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.529+0000 7f05aa407700 1 -- 172.21.15.38:0/91524174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a41973c0 con 0x7f05a41022a0 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.529+0000 7f05aa407700 1 --2- 172.21.15.38:0/91524174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a41022a0 0x7f05a4196e80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0594004fa0 tx=0x7f0594005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.530+0000 7f059b7fe700 1 -- 172.21.15.38:0/91524174 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0594027070 con 0x7f05a41022a0 2024-03-26T06:28:38.533 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.530+0000 7f05ac66b700 1 -- 172.21.15.38:0/91524174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a41976b0 con 0x7f05a41022a0 2024-03-26T06:28:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.530+0000 7f059b7fe700 1 -- 172.21.15.38:0/91524174 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f059400be40 con 0x7f05a41022a0 2024-03-26T06:28:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.530+0000 7f059b7fe700 1 -- 172.21.15.38:0/91524174 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f059400f460 con 0x7f05a41022a0 2024-03-26T06:28:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.530+0000 7f05ac66b700 1 -- 172.21.15.38:0/91524174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a419a340 con 0x7f05a41022a0 2024-03-26T06:28:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.531+0000 7f059b7fe700 1 -- 172.21.15.38:0/91524174 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f059402a070 con 0x7f05a41022a0 2024-03-26T06:28:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.531+0000 7f05ac66b700 1 -- 172.21.15.38:0/91524174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a4107fc0 con 0x7f05a41022a0 2024-03-26T06:28:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.531+0000 7f059b7fe700 1 --2- 172.21.15.38:0/91524174 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0590038fc0 0x7f059003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.531+0000 7f059b7fe700 1 -- 172.21.15.38:0/91524174 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f059401e030 con 0x7f05a41022a0 2024-03-26T06:28:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.531+0000 7f05a9c06700 1 --2- 172.21.15.38:0/91524174 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0590038fc0 0x7f059003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.531+0000 7f05a9c06700 1 --2- 172.21.15.38:0/91524174 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0590038fc0 0x7f059003b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f05a0006fd0 tx=0x7f05a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:38.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.540+0000 7f059b7fe700 1 -- 172.21.15.38:0/91524174 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0594013070 con 0x7f05a41022a0 2024-03-26T06:28:38.746 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.744+0000 7f05ac66b700 1 -- 172.21.15.38:0/91524174 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f05a402d9b0 con 0x7f05a41022a0 2024-03-26T06:28:38.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:38 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:38.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:38 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:38.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:38 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:28:38.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:38 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:38.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:38 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:38.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:38 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:38.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.747+0000 7f059b7fe700 1 -- 172.21.15.38:0/91524174 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f059400b960 con 0x7f05a41022a0 2024-03-26T06:28:38.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.750+0000 7f05ac66b700 1 -- 172.21.15.38:0/91524174 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0590038fc0 msgr2=0x7f059003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:38.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.750+0000 7f05ac66b700 1 --2- 172.21.15.38:0/91524174 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0590038fc0 0x7f059003b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f05a0006fd0 tx=0x7f05a0006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:38.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.750+0000 7f05ac66b700 1 -- 172.21.15.38:0/91524174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a41022a0 msgr2=0x7f05a4196e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:38.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.750+0000 7f05ac66b700 1 --2- 172.21.15.38:0/91524174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a41022a0 0x7f05a4196e80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0594004fa0 tx=0x7f0594005080 comp rx=0 tx=0).stop 2024-03-26T06:28:38.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.750+0000 7f05ac66b700 1 -- 172.21.15.38:0/91524174 shutdown_connections 2024-03-26T06:28:38.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.750+0000 7f05ac66b700 1 --2- 172.21.15.38:0/91524174 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0590038fc0 0x7f059003b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:38.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.750+0000 7f05ac66b700 1 --2- 172.21.15.38:0/91524174 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f05a41022a0 0x7f05a4196e80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:38.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.750+0000 7f05ac66b700 1 -- 172.21.15.38:0/91524174 >> 172.21.15.38:0/91524174 conn(0x7f05a40fbcb0 msgr2=0x7f05a40fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:38.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.750+0000 7f05ac66b700 1 -- 172.21.15.38:0/91524174 shutdown_connections 2024-03-26T06:28:38.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:38.750+0000 7f05ac66b700 1 -- 172.21.15.38:0/91524174 wait complete. 2024-03-26T06:28:39.390 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T06:28:39.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T06:28:39.630 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:28:40.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:39 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/91524174' entity='client.admin' 2024-03-26T06:28:40.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:40.039 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:41.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.189+0000 7fab58974700 1 -- 172.21.15.38:0/1808360800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab54102f10 msgr2=0x7fab54103330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:41.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.189+0000 7fab58974700 1 --2- 172.21.15.38:0/1808360800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab54102f10 0x7fab54103330 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fab44009910 tx=0x7fab44009c20 comp rx=0 tx=0).stop 2024-03-26T06:28:41.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.189+0000 7fab58974700 1 -- 172.21.15.38:0/1808360800 shutdown_connections 2024-03-26T06:28:41.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.189+0000 7fab58974700 1 --2- 172.21.15.38:0/1808360800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab54102f10 0x7fab54103330 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:41.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.189+0000 7fab58974700 1 -- 172.21.15.38:0/1808360800 >> 172.21.15.38:0/1808360800 conn(0x7fab540fcdd0 msgr2=0x7fab540ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:41.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.189+0000 7fab58974700 1 -- 172.21.15.38:0/1808360800 shutdown_connections 2024-03-26T06:28:41.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.189+0000 7fab58974700 1 -- 172.21.15.38:0/1808360800 wait complete. 2024-03-26T06:28:41.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.190+0000 7fab58974700 1 Processor -- start 2024-03-26T06:28:41.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.190+0000 7fab58974700 1 -- start start 2024-03-26T06:28:41.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab58974700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab54102f10 0x7fab54197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:41.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab58974700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab541079d0 con 0x7fab54102f10 2024-03-26T06:28:41.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab5259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab54102f10 0x7fab54197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab5259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab54102f10 0x7fab54197050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46832/0 (socket says 172.21.15.38:46832) 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab5259c700 1 -- 172.21.15.38:0/3044396096 learned_addr learned my addr 172.21.15.38:0/3044396096 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab5259c700 1 -- 172.21.15.38:0/3044396096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab54197590 con 0x7fab54102f10 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab5259c700 1 --2- 172.21.15.38:0/3044396096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab54102f10 0x7fab54197050 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fab44005850 tx=0x7fab44005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab437fe700 1 -- 172.21.15.38:0/3044396096 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab44026070 con 0x7fab54102f10 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab437fe700 1 -- 172.21.15.38:0/3044396096 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab440044a0 con 0x7fab54102f10 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab58974700 1 -- 172.21.15.38:0/3044396096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab54197820 con 0x7fab54102f10 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab437fe700 1 -- 172.21.15.38:0/3044396096 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab44003ab0 con 0x7fab54102f10 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.191+0000 7fab58974700 1 -- 172.21.15.38:0/3044396096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab5419a510 con 0x7fab54102f10 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.192+0000 7fab437fe700 1 -- 172.21.15.38:0/3044396096 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fab44029070 con 0x7fab54102f10 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.192+0000 7fab58974700 1 -- 172.21.15.38:0/3044396096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab54108150 con 0x7fab54102f10 2024-03-26T06:28:41.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.192+0000 7fab437fe700 1 --2- 172.21.15.38:0/3044396096 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fab3c039020 0x7fab3c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:41.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.192+0000 7fab437fe700 1 -- 172.21.15.38:0/3044396096 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fab4401e070 con 0x7fab54102f10 2024-03-26T06:28:41.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.192+0000 7fab51d9b700 1 --2- 172.21.15.38:0/3044396096 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fab3c039020 0x7fab3c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:41.196 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.193+0000 7fab51d9b700 1 --2- 172.21.15.38:0/3044396096 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fab3c039020 0x7fab3c03b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fab48006fd0 tx=0x7fab48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:41.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.198+0000 7fab437fe700 1 -- 172.21.15.38:0/3044396096 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fab44051070 con 0x7fab54102f10 2024-03-26T06:28:41.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.410+0000 7fab58974700 1 -- 172.21.15.38:0/3044396096 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fab5402d8a0 con 0x7fab3c039020 2024-03-26T06:28:41.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.413+0000 7fab437fe700 1 -- 172.21.15.38:0/3044396096 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fab5402d8a0 con 0x7fab3c039020 2024-03-26T06:28:41.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.415+0000 7fab58974700 1 -- 172.21.15.38:0/3044396096 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fab3c039020 msgr2=0x7fab3c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:41.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.415+0000 7fab58974700 1 --2- 172.21.15.38:0/3044396096 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fab3c039020 0x7fab3c03b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fab48006fd0 tx=0x7fab48006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:41.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.415+0000 7fab58974700 1 -- 172.21.15.38:0/3044396096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab54102f10 msgr2=0x7fab54197050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:41.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.415+0000 7fab58974700 1 --2- 172.21.15.38:0/3044396096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab54102f10 0x7fab54197050 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fab44005850 tx=0x7fab44005030 comp rx=0 tx=0).stop 2024-03-26T06:28:41.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.415+0000 7fab58974700 1 -- 172.21.15.38:0/3044396096 shutdown_connections 2024-03-26T06:28:41.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.415+0000 7fab58974700 1 --2- 172.21.15.38:0/3044396096 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fab3c039020 0x7fab3c03b4e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:41.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.415+0000 7fab58974700 1 --2- 172.21.15.38:0/3044396096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fab54102f10 0x7fab54197050 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:41.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.415+0000 7fab58974700 1 -- 172.21.15.38:0/3044396096 >> 172.21.15.38:0/3044396096 conn(0x7fab540fcdd0 msgr2=0x7fab540fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:41.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.415+0000 7fab58974700 1 -- 172.21.15.38:0/3044396096 shutdown_connections 2024-03-26T06:28:41.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:41.415+0000 7fab58974700 1 -- 172.21.15.38:0/3044396096 wait complete. 2024-03-26T06:28:42.156 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi121 2024-03-26T06:28:42.157 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-26T06:28:42.157 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/etc/ceph/ceph.conf 2024-03-26T06:28:42.176 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-26T06:28:42.176 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:28:42.234 INFO:tasks.cephadm:Adding host smithi121 to orchestrator... 2024-03-26T06:28:42.234 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch host add smithi121 2024-03-26T06:28:42.478 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:28:42.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:42.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:42.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:42.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:28:42.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:28:42.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T06:28:42.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: Updating smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:28:42.681 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:28:42.681 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: Updating smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:28:42.681 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:42.681 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:42.681 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:42 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:43.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.800+0000 7ff3e771f700 1 -- 172.21.15.38:0/4017988595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e01077c0 msgr2=0x7ff3e0107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:43.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.800+0000 7ff3e771f700 1 --2- 172.21.15.38:0/4017988595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e01077c0 0x7ff3e0107ba0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc009910 tx=0x7ff3dc009c20 comp rx=0 tx=0).stop 2024-03-26T06:28:43.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.800+0000 7ff3e771f700 1 -- 172.21.15.38:0/4017988595 shutdown_connections 2024-03-26T06:28:43.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.800+0000 7ff3e771f700 1 --2- 172.21.15.38:0/4017988595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e01077c0 0x7ff3e0107ba0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:43.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.800+0000 7ff3e771f700 1 -- 172.21.15.38:0/4017988595 >> 172.21.15.38:0/4017988595 conn(0x7ff3e00fd2f0 msgr2=0x7ff3e00ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:43.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.800+0000 7ff3e771f700 1 -- 172.21.15.38:0/4017988595 shutdown_connections 2024-03-26T06:28:43.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.800+0000 7ff3e771f700 1 -- 172.21.15.38:0/4017988595 wait complete. 2024-03-26T06:28:43.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.800+0000 7ff3e771f700 1 Processor -- start 2024-03-26T06:28:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.800+0000 7ff3e771f700 1 -- start start 2024-03-26T06:28:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3e771f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e01077c0 0x7ff3e0197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3e771f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e010be00 con 0x7ff3e01077c0 2024-03-26T06:28:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3e54bb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e01077c0 0x7ff3e0197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3e54bb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e01077c0 0x7ff3e0197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46862/0 (socket says 172.21.15.38:46862) 2024-03-26T06:28:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3e54bb700 1 -- 172.21.15.38:0/1265480607 learned_addr learned my addr 172.21.15.38:0/1265480607 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3e54bb700 1 -- 172.21.15.38:0/1265480607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e0197840 con 0x7ff3e01077c0 2024-03-26T06:28:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3e54bb700 1 --2- 172.21.15.38:0/1265480607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e01077c0 0x7ff3e0197300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc005850 tx=0x7ff3dc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3d67fc700 1 -- 172.21.15.38:0/1265480607 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3dc026070 con 0x7ff3e01077c0 2024-03-26T06:28:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3d67fc700 1 -- 172.21.15.38:0/1265480607 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3dc00bbd0 con 0x7ff3e01077c0 2024-03-26T06:28:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3e771f700 1 -- 172.21.15.38:0/1265480607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e0197b30 con 0x7ff3e01077c0 2024-03-26T06:28:43.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3d67fc700 1 -- 172.21.15.38:0/1265480607 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff3dc003970 con 0x7ff3e01077c0 2024-03-26T06:28:43.805 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.801+0000 7ff3e771f700 1 -- 172.21.15.38:0/1265480607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e019bbd0 con 0x7ff3e01077c0 2024-03-26T06:28:43.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.802+0000 7ff3d67fc700 1 -- 172.21.15.38:0/1265480607 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7ff3dc029070 con 0x7ff3e01077c0 2024-03-26T06:28:43.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.802+0000 7ff3d67fc700 1 --2- 172.21.15.38:0/1265480607 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff3cc039020 0x7ff3cc03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:43.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.802+0000 7ff3e771f700 1 -- 172.21.15.38:0/1265480607 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e0102f00 con 0x7ff3e01077c0 2024-03-26T06:28:43.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.802+0000 7ff3d67fc700 1 -- 172.21.15.38:0/1265480607 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff3dc01e070 con 0x7ff3e01077c0 2024-03-26T06:28:43.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.802+0000 7ff3e4cba700 1 --2- 172.21.15.38:0/1265480607 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff3cc039020 0x7ff3cc03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:43.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.803+0000 7ff3e4cba700 1 --2- 172.21.15.38:0/1265480607 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff3cc039020 0x7ff3cc03b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0006fd0 tx=0x7ff3d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:43.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:43.808+0000 7ff3d67fc700 1 -- 172.21.15.38:0/1265480607 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff3dc030020 con 0x7ff3e01077c0 2024-03-26T06:28:44.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:44.014+0000 7ff3e771f700 1 -- 172.21.15.38:0/1265480607 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi121", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3e0198660 con 0x7ff3cc039020 2024-03-26T06:28:44.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:44 smithi038 ceph-mon[32265]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi121", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:45.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:45 smithi038 ceph-mon[32265]: Deploying cephadm binary to smithi121 2024-03-26T06:28:47.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:47 smithi038 ceph-mon[32265]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:28:48.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:48.426+0000 7ff3d67fc700 1 -- 172.21.15.38:0/1265480607 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff3e0198660 con 0x7ff3cc039020 2024-03-26T06:28:48.428 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi121' with addr '172.21.15.121' 2024-03-26T06:28:48.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:48.428+0000 7ff3e771f700 1 -- 172.21.15.38:0/1265480607 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff3cc039020 msgr2=0x7ff3cc03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:48.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:48.428+0000 7ff3e771f700 1 --2- 172.21.15.38:0/1265480607 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff3cc039020 0x7ff3cc03b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0006fd0 tx=0x7ff3d0006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:48.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:48.428+0000 7ff3e771f700 1 -- 172.21.15.38:0/1265480607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e01077c0 msgr2=0x7ff3e0197300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:48.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:48.428+0000 7ff3e771f700 1 --2- 172.21.15.38:0/1265480607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e01077c0 0x7ff3e0197300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc005850 tx=0x7ff3dc005030 comp rx=0 tx=0).stop 2024-03-26T06:28:48.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:48.428+0000 7ff3e771f700 1 -- 172.21.15.38:0/1265480607 shutdown_connections 2024-03-26T06:28:48.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:48.428+0000 7ff3e771f700 1 --2- 172.21.15.38:0/1265480607 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff3cc039020 0x7ff3cc03b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:48.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:48.428+0000 7ff3e771f700 1 --2- 172.21.15.38:0/1265480607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff3e01077c0 0x7ff3e0197300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:48.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:48.428+0000 7ff3e771f700 1 -- 172.21.15.38:0/1265480607 >> 172.21.15.38:0/1265480607 conn(0x7ff3e00fd2f0 msgr2=0x7ff3e0109330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:48.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:48.429+0000 7ff3e771f700 1 -- 172.21.15.38:0/1265480607 shutdown_connections 2024-03-26T06:28:48.431 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:48.429+0000 7ff3e771f700 1 -- 172.21.15.38:0/1265480607 wait complete. 2024-03-26T06:28:49.037 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T06:28:49.275 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:28:49.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:49 smithi038 ceph-mon[32265]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:28:49.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:49.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:49 smithi038 ceph-mon[32265]: Added host smithi121 2024-03-26T06:28:49.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:49.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:49.712 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:50.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.791+0000 7f917e790700 1 -- 172.21.15.38:0/3923579703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9178102a10 msgr2=0x7f9178102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.791+0000 7f917e790700 1 --2- 172.21.15.38:0/3923579703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9178102a10 0x7f9178102df0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9168009910 tx=0x7f9168009c20 comp rx=0 tx=0).stop 2024-03-26T06:28:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.791+0000 7f917e790700 1 -- 172.21.15.38:0/3923579703 shutdown_connections 2024-03-26T06:28:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.791+0000 7f917e790700 1 --2- 172.21.15.38:0/3923579703 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9178102a10 0x7f9178102df0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.791+0000 7f917e790700 1 -- 172.21.15.38:0/3923579703 >> 172.21.15.38:0/3923579703 conn(0x7f9178077320 msgr2=0x7f91780fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.791+0000 7f917e790700 1 -- 172.21.15.38:0/3923579703 shutdown_connections 2024-03-26T06:28:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.791+0000 7f917e790700 1 -- 172.21.15.38:0/3923579703 wait complete. 2024-03-26T06:28:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.791+0000 7f917e790700 1 Processor -- start 2024-03-26T06:28:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.791+0000 7f917e790700 1 -- start start 2024-03-26T06:28:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.792+0000 7f917e790700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9178102a10 0x7f9178197390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.792+0000 7f917e790700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917810be60 con 0x7f9178102a10 2024-03-26T06:28:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.792+0000 7f9177fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9178102a10 0x7f9178197390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.792+0000 7f9177fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9178102a10 0x7f9178197390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55030/0 (socket says 172.21.15.38:55030) 2024-03-26T06:28:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.792+0000 7f9177fff700 1 -- 172.21.15.38:0/663847219 learned_addr learned my addr 172.21.15.38:0/663847219 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.792+0000 7f9177fff700 1 -- 172.21.15.38:0/663847219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91781978d0 con 0x7f9178102a10 2024-03-26T06:28:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.792+0000 7f9177fff700 1 --2- 172.21.15.38:0/663847219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9178102a10 0x7f9178197390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9168006010 tx=0x7f9168005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.792+0000 7f91757fa700 1 -- 172.21.15.38:0/663847219 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9168026070 con 0x7f9178102a10 2024-03-26T06:28:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.793+0000 7f91757fa700 1 -- 172.21.15.38:0/663847219 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f916800bc30 con 0x7f9178102a10 2024-03-26T06:28:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.793+0000 7f917e790700 1 -- 172.21.15.38:0/663847219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9178197b60 con 0x7f9178102a10 2024-03-26T06:28:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.793+0000 7f91757fa700 1 -- 172.21.15.38:0/663847219 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9168003ab0 con 0x7f9178102a10 2024-03-26T06:28:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.793+0000 7f917e790700 1 -- 172.21.15.38:0/663847219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9178197fc0 con 0x7f9178102a10 2024-03-26T06:28:50.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.793+0000 7f91757fa700 1 -- 172.21.15.38:0/663847219 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f9168029070 con 0x7f9178102a10 2024-03-26T06:28:50.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.793+0000 7f917e790700 1 -- 172.21.15.38:0/663847219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9178069a10 con 0x7f9178102a10 2024-03-26T06:28:50.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.794+0000 7f91757fa700 1 --2- 172.21.15.38:0/663847219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9160039020 0x7f916003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:50.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.794+0000 7f91757fa700 1 -- 172.21.15.38:0/663847219 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f916801e070 con 0x7f9178102a10 2024-03-26T06:28:50.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.794+0000 7f91777fe700 1 --2- 172.21.15.38:0/663847219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9160039020 0x7f916003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:50.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.794+0000 7f91777fe700 1 --2- 172.21.15.38:0/663847219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9160039020 0x7f916003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f916c006fd0 tx=0x7f916c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:50.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:50.799+0000 7f91757fa700 1 -- 172.21.15.38:0/663847219 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9168013070 con 0x7f9178102a10 2024-03-26T06:28:51.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.010+0000 7f917e790700 1 -- 172.21.15.38:0/663847219 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9178198760 con 0x7f9160039020 2024-03-26T06:28:51.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.011+0000 7f91757fa700 1 -- 172.21.15.38:0/663847219 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f9178198760 con 0x7f9160039020 2024-03-26T06:28:51.013 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:28:51.013 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}] 2024-03-26T06:28:51.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.013+0000 7f917e790700 1 -- 172.21.15.38:0/663847219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9160039020 msgr2=0x7f916003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:51.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.013+0000 7f917e790700 1 --2- 172.21.15.38:0/663847219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9160039020 0x7f916003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f916c006fd0 tx=0x7f916c006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:51.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.013+0000 7f917e790700 1 -- 172.21.15.38:0/663847219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9178102a10 msgr2=0x7f9178197390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:51.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.013+0000 7f917e790700 1 --2- 172.21.15.38:0/663847219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9178102a10 0x7f9178197390 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9168006010 tx=0x7f9168005050 comp rx=0 tx=0).stop 2024-03-26T06:28:51.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.013+0000 7f917e790700 1 -- 172.21.15.38:0/663847219 shutdown_connections 2024-03-26T06:28:51.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.013+0000 7f917e790700 1 --2- 172.21.15.38:0/663847219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9160039020 0x7f916003b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:51.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.013+0000 7f917e790700 1 --2- 172.21.15.38:0/663847219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9178102a10 0x7f9178197390 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:51.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.013+0000 7f917e790700 1 -- 172.21.15.38:0/663847219 >> 172.21.15.38:0/663847219 conn(0x7f9178077320 msgr2=0x7f91781037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:51.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.013+0000 7f917e790700 1 -- 172.21.15.38:0/663847219 shutdown_connections 2024-03-26T06:28:51.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:51.013+0000 7f917e790700 1 -- 172.21.15.38:0/663847219 wait complete. 2024-03-26T06:28:51.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:51 smithi038 ceph-mon[32265]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:28:51.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:51 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:51.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:51 smithi038 ceph-mon[32265]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:28:51.582 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi170 2024-03-26T06:28:51.583 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-26T06:28:51.583 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.conf 2024-03-26T06:28:51.602 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-26T06:28:51.603 DEBUG:teuthology.orchestra.run.smithi170:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:28:51.659 INFO:tasks.cephadm:Adding host smithi170 to orchestrator... 2024-03-26T06:28:51.660 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch host add smithi170 2024-03-26T06:28:51.895 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:28:53.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.323+0000 7f25a714e700 1 -- 172.21.15.38:0/1300689138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25a0107800 msgr2=0x7f25a0107be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:53.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.323+0000 7f25a714e700 1 --2- 172.21.15.38:0/1300689138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25a0107800 0x7f25a0107be0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2594009910 tx=0x7f2594009c20 comp rx=0 tx=0).stop 2024-03-26T06:28:53.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.323+0000 7f25a714e700 1 -- 172.21.15.38:0/1300689138 shutdown_connections 2024-03-26T06:28:53.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.323+0000 7f25a714e700 1 --2- 172.21.15.38:0/1300689138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25a0107800 0x7f25a0107be0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:53.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.323+0000 7f25a714e700 1 -- 172.21.15.38:0/1300689138 >> 172.21.15.38:0/1300689138 conn(0x7f25a0077ce0 msgr2=0x7f25a00780f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:53.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.323+0000 7f25a714e700 1 -- 172.21.15.38:0/1300689138 shutdown_connections 2024-03-26T06:28:53.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.323+0000 7f25a714e700 1 -- 172.21.15.38:0/1300689138 wait complete. 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.323+0000 7f25a714e700 1 Processor -- start 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.323+0000 7f25a714e700 1 -- start start 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.323+0000 7f25a714e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25a0107800 0x7f25a0199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.324+0000 7f25a714e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a010be20 con 0x7f25a0107800 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.324+0000 7f25a4eea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25a0107800 0x7f25a0199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.324+0000 7f25a4eea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25a0107800 0x7f25a0199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55046/0 (socket says 172.21.15.38:55046) 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.324+0000 7f25a4eea700 1 -- 172.21.15.38:0/3565763689 learned_addr learned my addr 172.21.15.38:0/3565763689 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.324+0000 7f25a4eea700 1 -- 172.21.15.38:0/3565763689 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a0199a40 con 0x7f25a0107800 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.324+0000 7f25a4eea700 1 --2- 172.21.15.38:0/3565763689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25a0107800 0x7f25a0199500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2594005850 tx=0x7f2594005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.324+0000 7f259dffb700 1 -- 172.21.15.38:0/3565763689 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2594026070 con 0x7f25a0107800 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.324+0000 7f259dffb700 1 -- 172.21.15.38:0/3565763689 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f259400bbd0 con 0x7f25a0107800 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.324+0000 7f25a714e700 1 -- 172.21.15.38:0/3565763689 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a0199d30 con 0x7f25a0107800 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.324+0000 7f259dffb700 1 -- 172.21.15.38:0/3565763689 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2594003970 con 0x7f25a0107800 2024-03-26T06:28:53.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.325+0000 7f25a714e700 1 -- 172.21.15.38:0/3565763689 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a019ddd0 con 0x7f25a0107800 2024-03-26T06:28:53.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.325+0000 7f259dffb700 1 -- 172.21.15.38:0/3565763689 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f2594029070 con 0x7f25a0107800 2024-03-26T06:28:53.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.325+0000 7f25a714e700 1 -- 172.21.15.38:0/3565763689 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a0069970 con 0x7f25a0107800 2024-03-26T06:28:53.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.326+0000 7f259dffb700 1 --2- 172.21.15.38:0/3565763689 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2590039020 0x7f259003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:53.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.326+0000 7f259dffb700 1 -- 172.21.15.38:0/3565763689 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f259401e070 con 0x7f25a0107800 2024-03-26T06:28:53.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.326+0000 7f259ffff700 1 --2- 172.21.15.38:0/3565763689 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2590039020 0x7f259003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:53.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.326+0000 7f259ffff700 1 --2- 172.21.15.38:0/3565763689 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2590039020 0x7f259003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f258c006fd0 tx=0x7f258c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.331+0000 7f259dffb700 1 -- 172.21.15.38:0/3565763689 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2594013070 con 0x7f25a0107800 2024-03-26T06:28:53.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:53.539+0000 7f25a714e700 1 -- 172.21.15.38:0/3565763689 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}) v1 -- 0x7f25a019a860 con 0x7f2590039020 2024-03-26T06:28:53.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:53 smithi038 ceph-mon[32265]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:28:54.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:54 smithi038 ceph-mon[32265]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi170", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:28:54.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:54 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:54.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:54 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:54.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:54 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:54.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:54 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:54.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:54 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi121", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:28:54.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:54 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:28:54.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:54 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:28:54.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:54 smithi038 ceph-mon[32265]: Updating smithi121:/etc/ceph/ceph.conf 2024-03-26T06:28:54.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:54 smithi038 ceph-mon[32265]: Updating smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:28:54.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:54 smithi038 ceph-mon[32265]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:28:56.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:55 smithi038 ceph-mon[32265]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:28:56.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:55 smithi038 ceph-mon[32265]: Deploying cephadm binary to smithi170 2024-03-26T06:28:56.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:55 smithi038 ceph-mon[32265]: Updating smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:28:56.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:56.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:56.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:57.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:57.802+0000 7f259dffb700 1 -- 172.21.15.38:0/3565763689 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f25a019a860 con 0x7f2590039020 2024-03-26T06:28:57.806 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi170' with addr '172.21.15.170' 2024-03-26T06:28:57.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:57.805+0000 7f25a714e700 1 -- 172.21.15.38:0/3565763689 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2590039020 msgr2=0x7f259003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:57.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:57.805+0000 7f25a714e700 1 --2- 172.21.15.38:0/3565763689 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2590039020 0x7f259003b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f258c006fd0 tx=0x7f258c006e40 comp rx=0 tx=0).stop 2024-03-26T06:28:57.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:57.805+0000 7f25a714e700 1 -- 172.21.15.38:0/3565763689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25a0107800 msgr2=0x7f25a0199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:57.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:57.805+0000 7f25a714e700 1 --2- 172.21.15.38:0/3565763689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25a0107800 0x7f25a0199500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2594005850 tx=0x7f2594005030 comp rx=0 tx=0).stop 2024-03-26T06:28:57.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:57.805+0000 7f25a714e700 1 -- 172.21.15.38:0/3565763689 shutdown_connections 2024-03-26T06:28:57.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:57.805+0000 7f25a714e700 1 --2- 172.21.15.38:0/3565763689 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f2590039020 0x7f259003b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:57.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:57.805+0000 7f25a714e700 1 --2- 172.21.15.38:0/3565763689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f25a0107800 0x7f25a0199500 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:57.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:57.805+0000 7f25a714e700 1 -- 172.21.15.38:0/3565763689 >> 172.21.15.38:0/3565763689 conn(0x7f25a0077ce0 msgr2=0x7f25a0102a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:57.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:57.805+0000 7f25a714e700 1 -- 172.21.15.38:0/3565763689 shutdown_connections 2024-03-26T06:28:57.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:57.805+0000 7f25a714e700 1 -- 172.21.15.38:0/3565763689 wait complete. 2024-03-26T06:28:58.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:57 smithi038 ceph-mon[32265]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:28:58.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T06:28:58.660 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:28:59.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:59.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:58 smithi038 ceph-mon[32265]: Added host smithi170 2024-03-26T06:28:59.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:28:59.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:59.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:28:59.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.981+0000 7f8cb5f45700 1 -- 172.21.15.38:0/174639799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cb01000b0 msgr2=0x7f8cb0100490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:28:59.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.981+0000 7f8cb5f45700 1 --2- 172.21.15.38:0/174639799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cb01000b0 0x7f8cb0100490 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009910 tx=0x7f8ca0009c20 comp rx=0 tx=0).stop 2024-03-26T06:28:59.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.981+0000 7f8cb5f45700 1 -- 172.21.15.38:0/174639799 shutdown_connections 2024-03-26T06:28:59.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.981+0000 7f8cb5f45700 1 --2- 172.21.15.38:0/174639799 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cb01000b0 0x7f8cb0100490 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.981+0000 7f8cb5f45700 1 -- 172.21.15.38:0/174639799 >> 172.21.15.38:0/174639799 conn(0x7f8cb0076d90 msgr2=0x7f8cb00771a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.981+0000 7f8cb5f45700 1 -- 172.21.15.38:0/174639799 shutdown_connections 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.981+0000 7f8cb5f45700 1 -- 172.21.15.38:0/174639799 wait complete. 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.981+0000 7f8cb5f45700 1 Processor -- start 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.981+0000 7f8cb5f45700 1 -- start start 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8cb5f45700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cb01000b0 0x7f8cb0195f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8cb5f45700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb010a480 con 0x7f8cb01000b0 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8caf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cb01000b0 0x7f8cb0195f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8caf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cb01000b0 0x7f8cb0195f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54980/0 (socket says 172.21.15.38:54980) 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8caf7fe700 1 -- 172.21.15.38:0/1795007548 learned_addr learned my addr 172.21.15.38:0/1795007548 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8caf7fe700 1 -- 172.21.15.38:0/1795007548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cb00fdb80 con 0x7f8cb01000b0 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8caf7fe700 1 --2- 172.21.15.38:0/1795007548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cb01000b0 0x7f8cb0195f00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0005850 tx=0x7f8ca0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:59.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8cacff9700 1 -- 172.21.15.38:0/1795007548 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ca0026070 con 0x7f8cb01000b0 2024-03-26T06:28:59.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8cb5f45700 1 -- 172.21.15.38:0/1795007548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb0196440 con 0x7f8cb01000b0 2024-03-26T06:28:59.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8cacff9700 1 -- 172.21.15.38:0/1795007548 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ca000bbd0 con 0x7f8cb01000b0 2024-03-26T06:28:59.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8cacff9700 1 -- 172.21.15.38:0/1795007548 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ca0003970 con 0x7f8cb01000b0 2024-03-26T06:28:59.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.982+0000 7f8cb5f45700 1 -- 172.21.15.38:0/1795007548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb0196900 con 0x7f8cb01000b0 2024-03-26T06:28:59.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.983+0000 7f8cacff9700 1 -- 172.21.15.38:0/1795007548 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f8ca0029070 con 0x7f8cb01000b0 2024-03-26T06:28:59.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.983+0000 7f8cb5f45700 1 -- 172.21.15.38:0/1795007548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb0101590 con 0x7f8cb01000b0 2024-03-26T06:28:59.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.983+0000 7f8cacff9700 1 --2- 172.21.15.38:0/1795007548 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8c98038fc0 0x7f8c9803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:28:59.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.984+0000 7f8cacff9700 1 -- 172.21.15.38:0/1795007548 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f8ca001e070 con 0x7f8cb01000b0 2024-03-26T06:28:59.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.984+0000 7f8caeffd700 1 --2- 172.21.15.38:0/1795007548 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8c98038fc0 0x7f8c9803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:28:59.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.984+0000 7f8caeffd700 1 --2- 172.21.15.38:0/1795007548 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8c98038fc0 0x7f8c9803b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4006fd0 tx=0x7f8ca4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:28:59.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:28:59.989+0000 7f8cacff9700 1 -- 172.21.15.38:0/1795007548 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8ca0013070 con 0x7f8cb01000b0 2024-03-26T06:29:00.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:28:59 smithi038 ceph-mon[32265]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:00.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.202+0000 7f8cb5f45700 1 -- 172.21.15.38:0/1795007548 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8cb00fd3e0 con 0x7f8c98038fc0 2024-03-26T06:29:00.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.203+0000 7f8cacff9700 1 -- 172.21.15.38:0/1795007548 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7f8cb00fd3e0 con 0x7f8c98038fc0 2024-03-26T06:29:00.206 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:29:00.206 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}, {"addr": "172.21.15.170", "hostname": "smithi170", "labels": [], "status": ""}] 2024-03-26T06:29:00.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.205+0000 7f8cb5f45700 1 -- 172.21.15.38:0/1795007548 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8c98038fc0 msgr2=0x7f8c9803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:00.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.205+0000 7f8cb5f45700 1 --2- 172.21.15.38:0/1795007548 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8c98038fc0 0x7f8c9803b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4006fd0 tx=0x7f8ca4006e40 comp rx=0 tx=0).stop 2024-03-26T06:29:00.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.205+0000 7f8cb5f45700 1 -- 172.21.15.38:0/1795007548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cb01000b0 msgr2=0x7f8cb0195f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:00.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.205+0000 7f8cb5f45700 1 --2- 172.21.15.38:0/1795007548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cb01000b0 0x7f8cb0195f00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0005850 tx=0x7f8ca0005030 comp rx=0 tx=0).stop 2024-03-26T06:29:00.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.205+0000 7f8cb5f45700 1 -- 172.21.15.38:0/1795007548 shutdown_connections 2024-03-26T06:29:00.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.205+0000 7f8cb5f45700 1 --2- 172.21.15.38:0/1795007548 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8c98038fc0 0x7f8c9803b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:00.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.205+0000 7f8cb5f45700 1 --2- 172.21.15.38:0/1795007548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8cb01000b0 0x7f8cb0195f00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:00.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.205+0000 7f8cb5f45700 1 -- 172.21.15.38:0/1795007548 >> 172.21.15.38:0/1795007548 conn(0x7f8cb0076d90 msgr2=0x7f8cb0107c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:00.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.205+0000 7f8cb5f45700 1 -- 172.21.15.38:0/1795007548 shutdown_connections 2024-03-26T06:29:00.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:00.205+0000 7f8cb5f45700 1 -- 172.21.15.38:0/1795007548 wait complete. 2024-03-26T06:29:00.821 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T06:29:00.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T06:29:01.057 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:29:01.429 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:01 smithi038 ceph-mon[32265]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:01.429 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:01 smithi038 ceph-mon[32265]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:29:01.429 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:02.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.619+0000 7fdb62505700 1 -- 172.21.15.38:0/3924324681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb5c1066b0 msgr2=0x7fdb5c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:02.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.619+0000 7fdb62505700 1 --2- 172.21.15.38:0/3924324681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb5c1066b0 0x7fdb5c106a90 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009910 tx=0x7fdb4c009c20 comp rx=0 tx=0).stop 2024-03-26T06:29:02.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.619+0000 7fdb62505700 1 -- 172.21.15.38:0/3924324681 shutdown_connections 2024-03-26T06:29:02.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.619+0000 7fdb62505700 1 --2- 172.21.15.38:0/3924324681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb5c1066b0 0x7fdb5c106a90 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:02.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.619+0000 7fdb62505700 1 -- 172.21.15.38:0/3924324681 >> 172.21.15.38:0/3924324681 conn(0x7fdb5c0fc1c0 msgr2=0x7fdb5c0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:02.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.619+0000 7fdb62505700 1 -- 172.21.15.38:0/3924324681 shutdown_connections 2024-03-26T06:29:02.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.619+0000 7fdb62505700 1 -- 172.21.15.38:0/3924324681 wait complete. 2024-03-26T06:29:02.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.620+0000 7fdb62505700 1 Processor -- start 2024-03-26T06:29:02.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.620+0000 7fdb62505700 1 -- start start 2024-03-26T06:29:02.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.620+0000 7fdb62505700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb5c1066b0 0x7fdb5c0fc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:02.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.620+0000 7fdb62505700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c10acf0 con 0x7fdb5c1066b0 2024-03-26T06:29:02.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.620+0000 7fdb5bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb5c1066b0 0x7fdb5c0fc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:02.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.620+0000 7fdb5bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb5c1066b0 0x7fdb5c0fc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55006/0 (socket says 172.21.15.38:55006) 2024-03-26T06:29:02.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.620+0000 7fdb5bfff700 1 -- 172.21.15.38:0/920997256 learned_addr learned my addr 172.21.15.38:0/920997256 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:29:02.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.620+0000 7fdb5bfff700 1 -- 172.21.15.38:0/920997256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0fe560 con 0x7fdb5c1066b0 2024-03-26T06:29:02.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.620+0000 7fdb5bfff700 1 --2- 172.21.15.38:0/920997256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb5c1066b0 0x7fdb5c0fc670 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c005850 tx=0x7fdb4c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:02.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.621+0000 7fdb597fa700 1 -- 172.21.15.38:0/920997256 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdb4c026070 con 0x7fdb5c1066b0 2024-03-26T06:29:02.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.621+0000 7fdb597fa700 1 -- 172.21.15.38:0/920997256 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb4c00bc30 con 0x7fdb5c1066b0 2024-03-26T06:29:02.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.621+0000 7fdb62505700 1 -- 172.21.15.38:0/920997256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c0fcc10 con 0x7fdb5c1066b0 2024-03-26T06:29:02.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.621+0000 7fdb597fa700 1 -- 172.21.15.38:0/920997256 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdb4c003a20 con 0x7fdb5c1066b0 2024-03-26T06:29:02.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.621+0000 7fdb62505700 1 -- 172.21.15.38:0/920997256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c0fd0d0 con 0x7fdb5c1066b0 2024-03-26T06:29:02.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.621+0000 7fdb597fa700 1 -- 172.21.15.38:0/920997256 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fdb4c029070 con 0x7fdb5c1066b0 2024-03-26T06:29:02.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.622+0000 7fdb62505700 1 -- 172.21.15.38:0/920997256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c101df0 con 0x7fdb5c1066b0 2024-03-26T06:29:02.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.622+0000 7fdb597fa700 1 --2- 172.21.15.38:0/920997256 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb48038fc0 0x7fdb4803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:02.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.622+0000 7fdb597fa700 1 -- 172.21.15.38:0/920997256 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdb4c01e070 con 0x7fdb5c1066b0 2024-03-26T06:29:02.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.622+0000 7fdb5b7fe700 1 --2- 172.21.15.38:0/920997256 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb48038fc0 0x7fdb4803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:02.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.622+0000 7fdb5b7fe700 1 --2- 172.21.15.38:0/920997256 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb48038fc0 0x7fdb4803b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdb44006fd0 tx=0x7fdb44006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:02.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.627+0000 7fdb597fa700 1 -- 172.21.15.38:0/920997256 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdb4c013070 con 0x7fdb5c1066b0 2024-03-26T06:29:02.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:02.837+0000 7fdb62505700 1 -- 172.21.15.38:0/920997256 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fdb5c0fdba0 con 0x7fdb5c1066b0 2024-03-26T06:29:03.339 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:03.337+0000 7fdb597fa700 1 -- 172.21.15.38:0/920997256 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fdb4c030020 con 0x7fdb5c1066b0 2024-03-26T06:29:03.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:03.339+0000 7fdb62505700 1 -- 172.21.15.38:0/920997256 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb48038fc0 msgr2=0x7fdb4803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:03.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:03.339+0000 7fdb62505700 1 --2- 172.21.15.38:0/920997256 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb48038fc0 0x7fdb4803b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdb44006fd0 tx=0x7fdb44006e40 comp rx=0 tx=0).stop 2024-03-26T06:29:03.341 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:03.339+0000 7fdb62505700 1 -- 172.21.15.38:0/920997256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb5c1066b0 msgr2=0x7fdb5c0fc670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:03.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:03.339+0000 7fdb62505700 1 --2- 172.21.15.38:0/920997256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb5c1066b0 0x7fdb5c0fc670 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c005850 tx=0x7fdb4c005030 comp rx=0 tx=0).stop 2024-03-26T06:29:03.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:03.340+0000 7fdb62505700 1 -- 172.21.15.38:0/920997256 shutdown_connections 2024-03-26T06:29:03.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:03.340+0000 7fdb62505700 1 --2- 172.21.15.38:0/920997256 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb48038fc0 0x7fdb4803b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:03.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:03.340+0000 7fdb62505700 1 --2- 172.21.15.38:0/920997256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb5c1066b0 0x7fdb5c0fc670 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:03.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:03.340+0000 7fdb62505700 1 -- 172.21.15.38:0/920997256 >> 172.21.15.38:0/920997256 conn(0x7fdb5c0fc1c0 msgr2=0x7fdb5c069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:03.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:03.340+0000 7fdb62505700 1 -- 172.21.15.38:0/920997256 shutdown_connections 2024-03-26T06:29:03.342 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:03.340+0000 7fdb62505700 1 -- 172.21.15.38:0/920997256 wait complete. 2024-03-26T06:29:03.342 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2024-03-26T06:29:03.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:03 smithi038 ceph-mon[32265]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:03.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:03 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/920997256' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T06:29:03.994 INFO:tasks.cephadm:Adding mon.a on smithi038 2024-03-26T06:29:03.995 INFO:tasks.cephadm:Adding mon.b on smithi121 2024-03-26T06:29:03.995 INFO:tasks.cephadm:Adding mon.c on smithi170 2024-03-26T06:29:03.995 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi038:172.21.15.38=a;smithi121:172.21.15.121=b;smithi170:172.21.15.170=c' 2024-03-26T06:29:04.231 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/920997256' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T06:29:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T06:29:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi170", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:29:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-26T06:29:04.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: Updating smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:04.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:29:04.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: Updating smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:29:04.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:04.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:04.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:05.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.664+0000 7f348be2a700 1 -- 172.21.15.170:0/1385656537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3484102a10 msgr2=0x7f3484102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:05.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.664+0000 7f348be2a700 1 --2- 172.21.15.170:0/1385656537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3484102a10 0x7f3484102df0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3480009910 tx=0x7f3480009c20 comp rx=0 tx=0).stop 2024-03-26T06:29:05.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.664+0000 7f348be2a700 1 -- 172.21.15.170:0/1385656537 shutdown_connections 2024-03-26T06:29:05.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.664+0000 7f348be2a700 1 --2- 172.21.15.170:0/1385656537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3484102a10 0x7f3484102df0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:05.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.664+0000 7f348be2a700 1 -- 172.21.15.170:0/1385656537 >> 172.21.15.170:0/1385656537 conn(0x7f34840fc180 msgr2=0x7f34840fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:05.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.665+0000 7f348be2a700 1 -- 172.21.15.170:0/1385656537 shutdown_connections 2024-03-26T06:29:05.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.665+0000 7f348be2a700 1 -- 172.21.15.170:0/1385656537 wait complete. 2024-03-26T06:29:05.668 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.665+0000 7f348be2a700 1 Processor -- start 2024-03-26T06:29:05.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.665+0000 7f348be2a700 1 -- start start 2024-03-26T06:29:05.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.665+0000 7f348be2a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3484102a10 0x7f3484100670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:05.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.665+0000 7f348be2a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348410be60 con 0x7f3484102a10 2024-03-26T06:29:05.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.666+0000 7f3489bc6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3484102a10 0x7f3484100670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:05.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.666+0000 7f3489bc6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3484102a10 0x7f3484100670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.170:54774/0 (socket says 172.21.15.170:54774) 2024-03-26T06:29:05.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.666+0000 7f3489bc6700 1 -- 172.21.15.170:0/3345579915 learned_addr learned my addr 172.21.15.170:0/3345579915 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-26T06:29:05.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.666+0000 7f3489bc6700 1 -- 172.21.15.170:0/3345579915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3484102560 con 0x7f3484102a10 2024-03-26T06:29:05.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.666+0000 7f3489bc6700 1 --2- 172.21.15.170:0/3345579915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3484102a10 0x7f3484100670 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3480006010 tx=0x7f3480005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:05.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.666+0000 7f347affd700 1 -- 172.21.15.170:0/3345579915 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3480026070 con 0x7f3484102a10 2024-03-26T06:29:05.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.666+0000 7f348be2a700 1 -- 172.21.15.170:0/3345579915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3484100bb0 con 0x7f3484102a10 2024-03-26T06:29:05.669 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.666+0000 7f347affd700 1 -- 172.21.15.170:0/3345579915 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f348000bc30 con 0x7f3484102a10 2024-03-26T06:29:05.670 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.666+0000 7f347affd700 1 -- 172.21.15.170:0/3345579915 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3480003ab0 con 0x7f3484102a10 2024-03-26T06:29:05.670 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.666+0000 7f348be2a700 1 -- 172.21.15.170:0/3345579915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3484101070 con 0x7f3484102a10 2024-03-26T06:29:05.671 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.667+0000 7f348be2a700 1 -- 172.21.15.170:0/3345579915 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348406a4c0 con 0x7f3484102a10 2024-03-26T06:29:05.671 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.667+0000 7f347affd700 1 -- 172.21.15.170:0/3345579915 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f3480029070 con 0x7f3484102a10 2024-03-26T06:29:05.671 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.667+0000 7f347affd700 1 --2- 172.21.15.170:0/3345579915 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3470038fc0 0x7f347003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:05.671 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.667+0000 7f347affd700 1 -- 172.21.15.170:0/3345579915 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f348001e070 con 0x7f3484102a10 2024-03-26T06:29:05.671 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.668+0000 7f34893c5700 1 --2- 172.21.15.170:0/3345579915 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3470038fc0 0x7f347003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:05.671 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.668+0000 7f34893c5700 1 --2- 172.21.15.170:0/3345579915 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3470038fc0 0x7f347003b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3474006fd0 tx=0x7f3474006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:05.675 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.673+0000 7f347affd700 1 -- 172.21.15.170:0/3345579915 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3480013070 con 0x7f3484102a10 2024-03-26T06:29:05.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:05 smithi038 ceph-mon[32265]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:05.877 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.875+0000 7f348be2a700 1 -- 172.21.15.170:0/3345579915 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi121:172.21.15.121=b;smithi170:172.21.15.170=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f3484101c60 con 0x7f3470038fc0 2024-03-26T06:29:05.885 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.884+0000 7f347affd700 1 -- 172.21.15.170:0/3345579915 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3484101c60 con 0x7f3470038fc0 2024-03-26T06:29:05.886 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mon update... 2024-03-26T06:29:05.887 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.886+0000 7f348be2a700 1 -- 172.21.15.170:0/3345579915 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3470038fc0 msgr2=0x7f347003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:05.887 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.886+0000 7f348be2a700 1 --2- 172.21.15.170:0/3345579915 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3470038fc0 0x7f347003b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3474006fd0 tx=0x7f3474006e40 comp rx=0 tx=0).stop 2024-03-26T06:29:05.888 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.886+0000 7f348be2a700 1 -- 172.21.15.170:0/3345579915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3484102a10 msgr2=0x7f3484100670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:05.888 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.886+0000 7f348be2a700 1 --2- 172.21.15.170:0/3345579915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3484102a10 0x7f3484100670 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3480006010 tx=0x7f3480005050 comp rx=0 tx=0).stop 2024-03-26T06:29:05.888 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.886+0000 7f348be2a700 1 -- 172.21.15.170:0/3345579915 shutdown_connections 2024-03-26T06:29:05.888 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.886+0000 7f348be2a700 1 --2- 172.21.15.170:0/3345579915 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3470038fc0 0x7f347003b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:05.888 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.886+0000 7f348be2a700 1 --2- 172.21.15.170:0/3345579915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3484102a10 0x7f3484100670 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:05.888 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.886+0000 7f348be2a700 1 -- 172.21.15.170:0/3345579915 >> 172.21.15.170:0/3345579915 conn(0x7f34840fc180 msgr2=0x7f34841037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:05.888 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.886+0000 7f348be2a700 1 -- 172.21.15.170:0/3345579915 shutdown_connections 2024-03-26T06:29:05.888 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:05.886+0000 7f348be2a700 1 -- 172.21.15.170:0/3345579915 wait complete. 2024-03-26T06:29:06.469 DEBUG:teuthology.orchestra.run.smithi121:mon.b> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T06:29:06.472 DEBUG:teuthology.orchestra.run.smithi170:mon.c> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T06:29:06.474 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T06:29:06.474 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T06:29:07.269 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:07.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:06 smithi038 ceph-mon[32265]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi121:172.21.15.121=b;smithi170:172.21.15.170=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:29:07.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:06 smithi038 ceph-mon[32265]: Saving service mon spec with placement smithi038:172.21.15.38=a;smithi121:172.21.15.121=b;smithi170:172.21.15.170=c;count:3 2024-03-26T06:29:07.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:06 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:07.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:06 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:07.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:06 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:07.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:06 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:07.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:06 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:07.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:06 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:07.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:06 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:07.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:06 smithi038 ceph-mon[32265]: Deploying daemon mon.c on smithi170 2024-03-26T06:29:08.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:07 smithi038 ceph-mon[32265]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:10.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:09 smithi038 ceph-mon[32265]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:10.492 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.488+0000 7f4fe108f700 1 -- 172.21.15.170:0/1609640544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fdc1066a0 msgr2=0x7f4fdc106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:10.492 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.488+0000 7f4fe108f700 1 --2- 172.21.15.170:0/1609640544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fdc1066a0 0x7f4fdc106a80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4009910 tx=0x7f4fd4009c20 comp rx=0 tx=0).stop 2024-03-26T06:29:10.492 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.488+0000 7f4fe108f700 1 -- 172.21.15.170:0/1609640544 shutdown_connections 2024-03-26T06:29:10.492 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.488+0000 7f4fe108f700 1 --2- 172.21.15.170:0/1609640544 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fdc1066a0 0x7f4fdc106a80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:10.492 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.488+0000 7f4fe108f700 1 -- 172.21.15.170:0/1609640544 >> 172.21.15.170:0/1609640544 conn(0x7f4fdc0fc190 msgr2=0x7f4fdc0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:10.493 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.488+0000 7f4fe108f700 1 -- 172.21.15.170:0/1609640544 shutdown_connections 2024-03-26T06:29:10.493 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.488+0000 7f4fe108f700 1 -- 172.21.15.170:0/1609640544 wait complete. 2024-03-26T06:29:10.493 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.489+0000 7f4fe108f700 1 Processor -- start 2024-03-26T06:29:10.493 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.489+0000 7f4fe108f700 1 -- start start 2024-03-26T06:29:10.493 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.489+0000 7f4fe108f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fdc1066a0 0x7f4fdc1961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:10.493 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.489+0000 7f4fe108f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fdc10ace0 con 0x7f4fdc1066a0 2024-03-26T06:29:10.493 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.490+0000 7f4fdad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fdc1066a0 0x7f4fdc1961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:10.493 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.490+0000 7f4fdad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fdc1066a0 0x7f4fdc1961d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.170:54788/0 (socket says 172.21.15.170:54788) 2024-03-26T06:29:10.493 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.490+0000 7f4fdad9d700 1 -- 172.21.15.170:0/668415754 learned_addr learned my addr 172.21.15.170:0/668415754 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-26T06:29:10.493 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.490+0000 7f4fdad9d700 1 -- 172.21.15.170:0/668415754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc196710 con 0x7f4fdc1066a0 2024-03-26T06:29:10.494 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.490+0000 7f4fdad9d700 1 --2- 172.21.15.170:0/668415754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fdc1066a0 0x7f4fdc1961d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4005850 tx=0x7f4fd4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:10.494 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.490+0000 7f4fc3fff700 1 -- 172.21.15.170:0/668415754 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4fd4026070 con 0x7f4fdc1066a0 2024-03-26T06:29:10.494 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.490+0000 7f4fc3fff700 1 -- 172.21.15.170:0/668415754 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4fd400bbd0 con 0x7f4fdc1066a0 2024-03-26T06:29:10.494 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.490+0000 7f4fe108f700 1 -- 172.21.15.170:0/668415754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fdc196a00 con 0x7f4fdc1066a0 2024-03-26T06:29:10.494 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.490+0000 7f4fc3fff700 1 -- 172.21.15.170:0/668415754 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4fd4003970 con 0x7f4fdc1066a0 2024-03-26T06:29:10.494 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.491+0000 7f4fe108f700 1 -- 172.21.15.170:0/668415754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fdc19aaa0 con 0x7f4fdc1066a0 2024-03-26T06:29:10.495 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.491+0000 7f4fc3fff700 1 -- 172.21.15.170:0/668415754 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f4fd4029070 con 0x7f4fdc1066a0 2024-03-26T06:29:10.495 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.491+0000 7f4fe108f700 1 -- 172.21.15.170:0/668415754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fdc101de0 con 0x7f4fdc1066a0 2024-03-26T06:29:10.495 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.492+0000 7f4fc3fff700 1 --2- 172.21.15.170:0/668415754 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4fc4038fc0 0x7f4fc403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:10.495 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.492+0000 7f4fc3fff700 1 -- 172.21.15.170:0/668415754 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4fd401e070 con 0x7f4fdc1066a0 2024-03-26T06:29:10.496 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.492+0000 7f4fda59c700 1 --2- 172.21.15.170:0/668415754 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4fc4038fc0 0x7f4fc403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:10.496 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.492+0000 7f4fda59c700 1 --2- 172.21.15.170:0/668415754 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4fc4038fc0 0x7f4fc403b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000ad30 tx=0x7f4fd00093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:10.499 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.497+0000 7f4fc3fff700 1 -- 172.21.15.170:0/668415754 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4fd4013070 con 0x7f4fdc1066a0 2024-03-26T06:29:10.780 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.778+0000 7f4fe108f700 1 -- 172.21.15.170:0/668415754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4fdc197530 con 0x7f4fdc1066a0 2024-03-26T06:29:10.780 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.779+0000 7f4fc3fff700 1 -- 172.21.15.170:0/668415754 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7f4fd4030090 con 0x7f4fdc1066a0 2024-03-26T06:29:10.781 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:29:10.781 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":1,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","modified":"2024-03-26T06:27:26.027721Z","created":"2024-03-26T06:27:26.027721Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T06:29:10.782 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.781+0000 7f4fe108f700 1 -- 172.21.15.170:0/668415754 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4fc4038fc0 msgr2=0x7f4fc403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:10.782 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.781+0000 7f4fe108f700 1 --2- 172.21.15.170:0/668415754 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4fc4038fc0 0x7f4fc403b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4fd000ad30 tx=0x7f4fd00093f0 comp rx=0 tx=0).stop 2024-03-26T06:29:10.783 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.781+0000 7f4fe108f700 1 -- 172.21.15.170:0/668415754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fdc1066a0 msgr2=0x7f4fdc1961d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:10.783 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.781+0000 7f4fe108f700 1 --2- 172.21.15.170:0/668415754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fdc1066a0 0x7f4fdc1961d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4fd4005850 tx=0x7f4fd4005030 comp rx=0 tx=0).stop 2024-03-26T06:29:10.783 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.781+0000 7f4fe108f700 1 -- 172.21.15.170:0/668415754 shutdown_connections 2024-03-26T06:29:10.783 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.781+0000 7f4fe108f700 1 --2- 172.21.15.170:0/668415754 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4fc4038fc0 0x7f4fc403b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:10.783 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.781+0000 7f4fe108f700 1 --2- 172.21.15.170:0/668415754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4fdc1066a0 0x7f4fdc1961d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:10.783 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.781+0000 7f4fe108f700 1 -- 172.21.15.170:0/668415754 >> 172.21.15.170:0/668415754 conn(0x7f4fdc0fc190 msgr2=0x7f4fdc108200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:10.783 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.781+0000 7f4fe108f700 1 -- 172.21.15.170:0/668415754 shutdown_connections 2024-03-26T06:29:10.783 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:10.781+0000 7f4fe108f700 1 -- 172.21.15.170:0/668415754 wait complete. 2024-03-26T06:29:10.783 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 1 2024-03-26T06:29:11.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:10 smithi170 ceph-mon[36912]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T06:29:11.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:11 smithi170 bash[36868]: bceeb49f7778be465de549d43e5e625e348adf89283b41f1ba6c0277a7184fdf 2024-03-26T06:29:11.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:11 smithi170 systemd[1]: Started Ceph mon.c for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:29:12.423 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T06:29:12.423 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T06:29:12.668 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:29:16.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:16.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:29:16.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: mon.a calling monitor election 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: mon.c calling monitor election 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: monmap e2: 2 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: fsmap 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T06:29:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: mgrmap e13: x(active, since 49s) 2024-03-26T06:29:16.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: overall HEALTH_OK 2024-03-26T06:29:16.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:16.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:16.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:16.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:16.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:16.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:16.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:29:16.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: mon.a calling monitor election 2024-03-26T06:29:16.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:16.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: mon.c calling monitor election 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: monmap e2: 2 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: fsmap 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: mgrmap e13: x(active, since 49s) 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: overall HEALTH_OK 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:16.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:17.189 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.186+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/2079716324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd541067d0 msgr2=0x7fbd3c005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:17.190 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.186+0000 7fbd5a2b6700 1 --2- 172.21.15.170:0/2079716324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd541067d0 0x7fbd3c005610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbd44005fa0 tx=0x7fbd4400ffc0 comp rx=0 tx=0).stop 2024-03-26T06:29:17.190 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.186+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/2079716324 shutdown_connections 2024-03-26T06:29:17.190 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.186+0000 7fbd5a2b6700 1 --2- 172.21.15.170:0/2079716324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd541067d0 0x7fbd3c005610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:17.190 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.186+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/2079716324 >> 172.21.15.170:0/2079716324 conn(0x7fbd540fc2e0 msgr2=0x7fbd540fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:17.190 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.186+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/2079716324 shutdown_connections 2024-03-26T06:29:17.190 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.186+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/2079716324 wait complete. 2024-03-26T06:29:17.190 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.187+0000 7fbd5a2b6700 1 Processor -- start 2024-03-26T06:29:17.191 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.187+0000 7fbd5a2b6700 1 -- start start 2024-03-26T06:29:17.192 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.187+0000 7fbd5a2b6700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd541067d0 0x7fbd541023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:17.192 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.187+0000 7fbd5a2b6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd541028e0 0x7fbd54102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:17.192 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.187+0000 7fbd5a2b6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd54108870 con 0x7fbd541028e0 2024-03-26T06:29:17.193 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.187+0000 7fbd5a2b6700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd541086f0 con 0x7fbd541067d0 2024-03-26T06:29:17.193 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd53fff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd541067d0 0x7fbd541023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:17.193 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd53fff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd541067d0 0x7fbd541023a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:60104/0 (socket says 172.21.15.170:60104) 2024-03-26T06:29:17.193 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd53fff700 1 -- 172.21.15.170:0/1200614020 learned_addr learned my addr 172.21.15.170:0/1200614020 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-26T06:29:17.193 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd53fff700 1 -- 172.21.15.170:0/1200614020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd541067d0 msgr2=0x7fbd541023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-26T06:29:17.193 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd53fff700 1 -- 172.21.15.170:0/1200614020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd541067d0 msgr2=0x7fbd541023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-26T06:29:17.193 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd53fff700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd541067d0 0x7fbd541023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-26T06:29:17.193 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd537fe700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd541028e0 0x7fbd54102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:17.193 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd53fff700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd541067d0 0x7fbd541023a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T06:29:17.194 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd537fe700 1 -- 172.21.15.170:0/1200614020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd541067d0 msgr2=0x7fbd541023a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:29:17.194 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd537fe700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd541067d0 0x7fbd541023a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:17.194 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd537fe700 1 -- 172.21.15.170:0/1200614020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd54104270 con 0x7fbd541028e0 2024-03-26T06:29:17.194 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.188+0000 7fbd537fe700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd541028e0 0x7fbd54102d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800b4d0 tx=0x7fbd4800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:17.194 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.189+0000 7fbd517fa700 1 -- 172.21.15.170:0/1200614020 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fbd48011630 con 0x7fbd541028e0 2024-03-26T06:29:17.194 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.189+0000 7fbd517fa700 1 -- 172.21.15.170:0/1200614020 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd48011cb0 con 0x7fbd541028e0 2024-03-26T06:29:17.194 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.189+0000 7fbd517fa700 1 -- 172.21.15.170:0/1200614020 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fbd48010e60 con 0x7fbd541028e0 2024-03-26T06:29:17.194 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.189+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/1200614020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd541a3690 con 0x7fbd541028e0 2024-03-26T06:29:17.194 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.189+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/1200614020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd541a3bf0 con 0x7fbd541028e0 2024-03-26T06:29:17.194 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.190+0000 7fbd517fa700 1 -- 172.21.15.170:0/1200614020 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fbd48018020 con 0x7fbd541028e0 2024-03-26T06:29:17.195 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.190+0000 7fbd517fa700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbd40038d00 0x7fbd4003b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:17.195 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.190+0000 7fbd517fa700 1 -- 172.21.15.170:0/1200614020 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fbd4804ca70 con 0x7fbd541028e0 2024-03-26T06:29:17.195 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.190+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/1200614020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd54101f10 con 0x7fbd541028e0 2024-03-26T06:29:17.195 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.190+0000 7fbd53fff700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbd40038d00 0x7fbd4003b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:17.196 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.191+0000 7fbd53fff700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbd40038d00 0x7fbd4003b1c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd44005fd0 tx=0x7fbd4401d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:17.198 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.196+0000 7fbd517fa700 1 -- 172.21.15.170:0/1200614020 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbd4801f020 con 0x7fbd541028e0 2024-03-26T06:29:17.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:16 smithi038 ceph-mon[32265]: Deploying daemon mon.b on smithi121 2024-03-26T06:29:17.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:16 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:17.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:16 smithi170 ceph-mon[36912]: Deploying daemon mon.b on smithi121 2024-03-26T06:29:17.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:16 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:17.474 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.471+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/1200614020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd541a3f60 con 0x7fbd541028e0 2024-03-26T06:29:17.474 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.472+0000 7fbd517fa700 1 -- 172.21.15.170:0/1200614020 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7fbd48018310 con 0x7fbd541028e0 2024-03-26T06:29:17.475 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:29:17.475 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":2,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","modified":"2024-03-26T06:29:10.851177Z","created":"2024-03-26T06:27:26.027721Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T06:29:17.477 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.474+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/1200614020 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbd40038d00 msgr2=0x7fbd4003b1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:17.477 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.474+0000 7fbd5a2b6700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbd40038d00 0x7fbd4003b1c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd44005fd0 tx=0x7fbd4401d040 comp rx=0 tx=0).stop 2024-03-26T06:29:17.477 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.474+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/1200614020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd541028e0 msgr2=0x7fbd54102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:17.477 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.474+0000 7fbd5a2b6700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd541028e0 0x7fbd54102d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800b4d0 tx=0x7fbd4800b890 comp rx=0 tx=0).stop 2024-03-26T06:29:17.477 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.474+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/1200614020 shutdown_connections 2024-03-26T06:29:17.477 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.474+0000 7fbd5a2b6700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbd40038d00 0x7fbd4003b1c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:17.478 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.474+0000 7fbd5a2b6700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbd541067d0 0x7fbd541023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:17.478 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.474+0000 7fbd5a2b6700 1 --2- 172.21.15.170:0/1200614020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd541028e0 0x7fbd54102d60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:17.478 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.474+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/1200614020 >> 172.21.15.170:0/1200614020 conn(0x7fbd540fc2e0 msgr2=0x7fbd5406a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:17.478 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.474+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/1200614020 shutdown_connections 2024-03-26T06:29:17.478 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:17.474+0000 7fbd5a2b6700 1 -- 172.21.15.170:0/1200614020 wait complete. 2024-03-26T06:29:17.479 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 2 2024-03-26T06:29:18.075 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:17 smithi170 ceph-mon[36912]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:18.075 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:17 smithi170 ceph-mon[36912]: from='client.? 172.21.15.170:0/1200614020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T06:29:18.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:17 smithi038 ceph-mon[32265]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:18.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:17 smithi038 ceph-mon[32265]: from='client.? 172.21.15.170:0/1200614020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T06:29:19.104 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T06:29:19.104 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T06:29:19.347 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:29:20.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:19 smithi121 systemd[1]: Started Ceph mon.b for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:29:25.163 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.161+0000 7f0ed8689700 1 -- 172.21.15.170:0/4165173067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ebc0035c0 msgr2=0x7f0ebc005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:25.163 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.161+0000 7f0ed8689700 1 --2- 172.21.15.170:0/4165173067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ebc0035c0 0x7f0ebc005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0101090 tx=0x7f0ec000fe60 comp rx=0 tx=0).stop 2024-03-26T06:29:25.163 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.161+0000 7f0ed8689700 1 -- 172.21.15.170:0/4165173067 shutdown_connections 2024-03-26T06:29:25.163 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.161+0000 7f0ed8689700 1 --2- 172.21.15.170:0/4165173067 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ebc0035c0 0x7f0ebc005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:25.163 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.161+0000 7f0ed8689700 1 -- 172.21.15.170:0/4165173067 >> 172.21.15.170:0/4165173067 conn(0x7f0ed0076dd0 msgr2=0x7f0ed00771e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:25.163 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.161+0000 7f0ed8689700 1 -- 172.21.15.170:0/4165173067 shutdown_connections 2024-03-26T06:29:25.164 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.161+0000 7f0ed8689700 1 -- 172.21.15.170:0/4165173067 wait complete. 2024-03-26T06:29:25.166 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.161+0000 7f0ed8689700 1 Processor -- start 2024-03-26T06:29:25.166 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed8689700 1 -- start start 2024-03-26T06:29:25.166 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed8689700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ebc0035c0 0x7f0ed01011d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:25.166 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed8689700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ed0101710 0x7f0ed0101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:25.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed8689700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed01a3740 0x7f0ed01a5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:25.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed8689700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed0109990 con 0x7f0ed01a3740 2024-03-26T06:29:25.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed8689700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed0109810 con 0x7f0ebc0035c0 2024-03-26T06:29:25.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed8689700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed0109b10 con 0x7f0ed0101710 2024-03-26T06:29:25.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed5c24700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ed0101710 0x7f0ed0101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:25.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed5c24700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ed0101710 0x7f0ed0101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:60112/0 (socket says 172.21.15.170:60112) 2024-03-26T06:29:25.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed6425700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ebc0035c0 0x7f0ed01011d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:25.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed5c24700 1 -- 172.21.15.170:0/1903611822 learned_addr learned my addr 172.21.15.170:0/1903611822 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-26T06:29:25.167 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed6c26700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed01a3740 0x7f0ed01a5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed5c24700 1 -- 172.21.15.170:0/1903611822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ed0101710 msgr2=0x7f0ed0101b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed5c24700 1 -- 172.21.15.170:0/1903611822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ed0101710 msgr2=0x7f0ed0101b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed5c24700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ed0101710 0x7f0ed0101b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed6425700 1 -- 172.21.15.170:0/1903611822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ebc0035c0 msgr2=0x7f0ed01011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed6425700 1 -- 172.21.15.170:0/1903611822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ebc0035c0 msgr2=0x7f0ed01011d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed6425700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ebc0035c0 0x7f0ed01011d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed5c24700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ed0101710 0x7f0ed0101b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.162+0000 7f0ed6425700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ebc0035c0 0x7f0ed01011d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.163+0000 7f0ed6c26700 1 -- 172.21.15.170:0/1903611822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ed0101710 msgr2=0x7f0ed0101b90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.163+0000 7f0ed6c26700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ed0101710 0x7f0ed0101b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.163+0000 7f0ed6c26700 1 -- 172.21.15.170:0/1903611822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ebc0035c0 msgr2=0x7f0ed01011d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:29:25.168 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.163+0000 7f0ed6c26700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ebc0035c0 0x7f0ed01011d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.163+0000 7f0ed6c26700 1 -- 172.21.15.170:0/1903611822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed01a6260 con 0x7f0ed01a3740 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.163+0000 7f0ed6c26700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed01a3740 0x7f0ed01a5b30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800bd50 tx=0x7f0ec800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.163+0000 7f0ec77fe700 1 -- 172.21.15.170:0/1903611822 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8019070 con 0x7f0ed01a3740 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.163+0000 7f0ec77fe700 1 -- 172.21.15.170:0/1903611822 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ec800cd10 con 0x7f0ed01a3740 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.163+0000 7f0ec77fe700 1 -- 172.21.15.170:0/1903611822 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec80127e0 con 0x7f0ed01a3740 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.163+0000 7f0ed8689700 1 -- 172.21.15.170:0/1903611822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed01a6550 con 0x7f0ed01a3740 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.163+0000 7f0ed8689700 1 -- 172.21.15.170:0/1903611822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed01a6a90 con 0x7f0ed01a3740 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.164+0000 7f0ec77fe700 1 -- 172.21.15.170:0/1903611822 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f0ec801c070 con 0x7f0ed01a3740 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.164+0000 7f0ed8689700 1 -- 172.21.15.170:0/1903611822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed0063410 con 0x7f0ed01a3740 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.164+0000 7f0ec77fe700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0eb4039450 0x7f0eb403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.165+0000 7f0ec77fe700 1 -- 172.21.15.170:0/1903611822 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0ec804b9a0 con 0x7f0ed01a3740 2024-03-26T06:29:25.169 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.165+0000 7f0ed6425700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0eb4039450 0x7f0eb403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:25.170 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.165+0000 7f0ed6425700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0eb4039450 0x7f0eb403b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0020040 tx=0x7f0ec0001e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:25.171 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.170+0000 7f0ec77fe700 1 -- 172.21.15.170:0/1903611822 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0ec8013c30 con 0x7f0ed01a3740 2024-03-26T06:29:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:29:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: mon.c calling monitor election 2024-03-26T06:29:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: mon.a calling monitor election 2024-03-26T06:29:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:25.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:25.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T06:29:25.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T06:29:25.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: fsmap 2024-03-26T06:29:25.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T06:29:25.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: mgrmap e13: x(active, since 58s) 2024-03-26T06:29:25.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T06:29:25.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T06:29:25.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T06:29:25.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2024-03-26T06:29:25.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:25.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:29:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: mon.c calling monitor election 2024-03-26T06:29:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: mon.a calling monitor election 2024-03-26T06:29:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: fsmap 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: mgrmap e13: x(active, since 58s) 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T06:29:25.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T06:29:25.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T06:29:25.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2024-03-26T06:29:25.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:25.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:25.472 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.471+0000 7f0ed8689700 1 -- 172.21.15.170:0/1903611822 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0ed01a6e20 con 0x7f0ed01a3740 2024-03-26T06:29:25.473 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.471+0000 7f0ec77fe700 1 -- 172.21.15.170:0/1903611822 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1283 (secure 0 0 0) 0x7f0ec800e090 con 0x7f0ed01a3740 2024-03-26T06:29:25.473 INFO:teuthology.orchestra.run.smithi170.stdout: 2024-03-26T06:29:25.473 INFO:teuthology.orchestra.run.smithi170.stdout:{"epoch":3,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","modified":"2024-03-26T06:29:19.948945Z","created":"2024-03-26T06:27:26.027721Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:3300","nonce":0},{"type":"v1","addr":"172.21.15.170:6789","nonce":0}]},"addr":"172.21.15.170:6789/0","public_addr":"172.21.15.170:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T06:29:25.475 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 -- 172.21.15.170:0/1903611822 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0eb4039450 msgr2=0x7f0eb403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:25.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0eb4039450 0x7f0eb403b910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0ec0020040 tx=0x7f0ec0001e70 comp rx=0 tx=0).stop 2024-03-26T06:29:25.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 -- 172.21.15.170:0/1903611822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed01a3740 msgr2=0x7f0ed01a5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:25.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed01a3740 0x7f0ed01a5b30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800bd50 tx=0x7f0ec800be30 comp rx=0 tx=0).stop 2024-03-26T06:29:25.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 -- 172.21.15.170:0/1903611822 shutdown_connections 2024-03-26T06:29:25.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ed01a3740 0x7f0ed01a5b30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:25.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0eb4039450 0x7f0eb403b910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:25.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ebc0035c0 0x7f0ed01011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:25.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 --2- 172.21.15.170:0/1903611822 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0ed0101710 0x7f0ed0101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:25.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 -- 172.21.15.170:0/1903611822 >> 172.21.15.170:0/1903611822 conn(0x7f0ed0076dd0 msgr2=0x7f0ed00ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:25.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 -- 172.21.15.170:0/1903611822 shutdown_connections 2024-03-26T06:29:25.476 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:25.474+0000 7f0ed8689700 1 -- 172.21.15.170:0/1903611822 wait complete. 2024-03-26T06:29:25.477 INFO:teuthology.orchestra.run.smithi170.stderr:dumped monmap epoch 3 2024-03-26T06:29:26.105 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T06:29:26.105 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T06:29:26.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:26 smithi038 ceph-mon[32265]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:26 smithi038 ceph-mon[32265]: from='client.? 172.21.15.170:0/1903611822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T06:29:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:26.360 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:29:26.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:26 smithi170 ceph-mon[36912]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:26.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:26.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:26.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:26.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:26 smithi170 ceph-mon[36912]: from='client.? 172.21.15.170:0/1903611822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T06:29:26.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:27.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: mon.c calling monitor election 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: mon.a calling monitor election 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: fsmap 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: mgrmap e13: x(active, since 58s) 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: mon.b (rank 2) addr [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] is down (out of quorum) 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:27.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='client.? 172.21.15.170:0/1903611822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T06:29:27.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:28.012 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: mon.b calling monitor election 2024-03-26T06:29:28.012 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:28.012 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T06:29:28.012 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: Updating smithi121:/etc/ceph/ceph.conf 2024-03-26T06:29:28.012 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-26T06:29:28.012 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: mon.b calling monitor election 2024-03-26T06:29:28.012 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: mon.a calling monitor election 2024-03-26T06:29:28.012 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: mon.c calling monitor election 2024-03-26T06:29:28.012 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T06:29:28.012 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: fsmap 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: mgrmap e13: x(active, since 60s) 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: Cluster is now healthy 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: overall HEALTH_OK 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: Updating smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: Updating smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: Updating smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.013 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:28.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T06:29:28.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:28.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.115+0000 7f35e603d700 1 -- 172.21.15.38:0/1078629315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e01066c0 msgr2=0x7f35e0106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:28.119 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.115+0000 7f35e603d700 1 --2- 172.21.15.38:0/1078629315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e01066c0 0x7f35e0106aa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009910 tx=0x7f35d0009c20 comp rx=0 tx=0).stop 2024-03-26T06:29:28.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.115+0000 7f35e603d700 1 -- 172.21.15.38:0/1078629315 shutdown_connections 2024-03-26T06:29:28.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.115+0000 7f35e603d700 1 --2- 172.21.15.38:0/1078629315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e01066c0 0x7f35e0106aa0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:28.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.115+0000 7f35e603d700 1 -- 172.21.15.38:0/1078629315 >> 172.21.15.38:0/1078629315 conn(0x7f35e00fc1b0 msgr2=0x7f35e00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:28.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.115+0000 7f35e603d700 1 -- 172.21.15.38:0/1078629315 shutdown_connections 2024-03-26T06:29:28.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.115+0000 7f35e603d700 1 -- 172.21.15.38:0/1078629315 wait complete. 2024-03-26T06:29:28.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.116+0000 7f35e603d700 1 Processor -- start 2024-03-26T06:29:28.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.116+0000 7f35e603d700 1 -- start start 2024-03-26T06:29:28.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.116+0000 7f35e603d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35e01066c0 0x7f35e01983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:28.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.116+0000 7f35e603d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0198900 0x7f35e0198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:28.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.116+0000 7f35e603d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e01a1430 0x7f35e01a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:28.120 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.116+0000 7f35e603d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e010ad00 con 0x7f35e0198900 2024-03-26T06:29:28.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.116+0000 7f35e603d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e010ab80 con 0x7f35e01a1430 2024-03-26T06:29:28.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.116+0000 7f35e603d700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e010ae80 con 0x7f35e01066c0 2024-03-26T06:29:28.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.116+0000 7f35deffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0198900 0x7f35e0198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:28.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.116+0000 7f35deffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0198900 0x7f35e0198d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44318/0 (socket says 172.21.15.38:44318) 2024-03-26T06:29:28.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.116+0000 7f35deffd700 1 -- 172.21.15.38:0/1718729372 learned_addr learned my addr 172.21.15.38:0/1718729372 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:29:28.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35dffff700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e01a1430 0x7f35e01a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:28.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35df7fe700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35e01066c0 0x7f35e01983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:28.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35deffd700 1 -- 172.21.15.38:0/1718729372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35e01066c0 msgr2=0x7f35e01983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:28.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35deffd700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35e01066c0 0x7f35e01983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:28.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35deffd700 1 -- 172.21.15.38:0/1718729372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e01a1430 msgr2=0x7f35e01a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:28.121 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35deffd700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e01a1430 0x7f35e01a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:28.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35deffd700 1 -- 172.21.15.38:0/1718729372 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e01a4030 con 0x7f35e0198900 2024-03-26T06:29:28.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35deffd700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0198900 0x7f35e0198d80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f35d400b780 tx=0x7f35d400ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:28.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35dcff9700 1 -- 172.21.15.38:0/1718729372 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4004230 con 0x7f35e0198900 2024-03-26T06:29:28.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35dcff9700 1 -- 172.21.15.38:0/1718729372 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35d40043d0 con 0x7f35e0198900 2024-03-26T06:29:28.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35e603d700 1 -- 172.21.15.38:0/1718729372 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e01a4320 con 0x7f35e0198900 2024-03-26T06:29:28.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35dcff9700 1 -- 172.21.15.38:0/1718729372 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4011c90 con 0x7f35e0198900 2024-03-26T06:29:28.122 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.117+0000 7f35e603d700 1 -- 172.21.15.38:0/1718729372 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e01a4860 con 0x7f35e0198900 2024-03-26T06:29:28.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.118+0000 7f35dcff9700 1 -- 172.21.15.38:0/1718729372 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f35d401a400 con 0x7f35e0198900 2024-03-26T06:29:28.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.118+0000 7f35e603d700 1 -- 172.21.15.38:0/1718729372 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e0063410 con 0x7f35e0198900 2024-03-26T06:29:28.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.118+0000 7f35dcff9700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f35c8039450 0x7f35c803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:28.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.118+0000 7f35dcff9700 1 -- 172.21.15.38:0/1718729372 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f35d404bb50 con 0x7f35e0198900 2024-03-26T06:29:28.123 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.118+0000 7f35df7fe700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f35c8039450 0x7f35c803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:28.124 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.119+0000 7f35df7fe700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f35c8039450 0x7f35c803b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009750 tx=0x7f35d0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:28.125 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.123+0000 7f35dcff9700 1 -- 172.21.15.38:0/1718729372 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f35d4020500 con 0x7f35e0198900 2024-03-26T06:29:28.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: mon.b calling monitor election 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: Updating smithi121:/etc/ceph/ceph.conf 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: mon.b calling monitor election 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: mon.a calling monitor election 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: mon.c calling monitor election 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: fsmap 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: mgrmap e13: x(active, since 60s) 2024-03-26T06:29:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: Cluster is now healthy 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: overall HEALTH_OK 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: Updating smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: Updating smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: Updating smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:28.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T06:29:28.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: mon.b calling monitor election 2024-03-26T06:29:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T06:29:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: Updating smithi121:/etc/ceph/ceph.conf 2024-03-26T06:29:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-26T06:29:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: mon.b calling monitor election 2024-03-26T06:29:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: mon.a calling monitor election 2024-03-26T06:29:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: mon.c calling monitor election 2024-03-26T06:29:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T06:29:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0],c=[v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T06:29:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: fsmap 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: mgrmap e13: x(active, since 60s) 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: Cluster is now healthy 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: overall HEALTH_OK 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: Updating smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: Updating smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: Updating smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.170 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.170 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.170 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.170 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.170 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.170 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.170 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:28.170 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:28.170 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T06:29:28.170 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:28.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.332+0000 7f35e603d700 1 -- 172.21.15.38:0/1718729372 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f35e01a4bf0 con 0x7f35e0198900 2024-03-26T06:29:28.335 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.333+0000 7f35dcff9700 1 -- 172.21.15.38:0/1718729372 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f35d40202a0 con 0x7f35e0198900 2024-03-26T06:29:28.335 INFO:teuthology.orchestra.run.smithi038.stdout:# minimal ceph.conf for c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:29:28.335 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2024-03-26T06:29:28.335 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:29:28.335 INFO:teuthology.orchestra.run.smithi038.stdout: mon_host = [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] 2024-03-26T06:29:28.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 -- 172.21.15.38:0/1718729372 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f35c8039450 msgr2=0x7f35c803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:28.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f35c8039450 0x7f35c803b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f35d0009750 tx=0x7f35d0005fb0 comp rx=0 tx=0).stop 2024-03-26T06:29:28.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 -- 172.21.15.38:0/1718729372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0198900 msgr2=0x7f35e0198d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:28.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0198900 0x7f35e0198d80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f35d400b780 tx=0x7f35d400ba90 comp rx=0 tx=0).stop 2024-03-26T06:29:28.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 -- 172.21.15.38:0/1718729372 shutdown_connections 2024-03-26T06:29:28.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f35c8039450 0x7f35c803b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:28.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f35e01a1430 0x7f35e01a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:28.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f35e01066c0 0x7f35e01983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:28.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 --2- 172.21.15.38:0/1718729372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35e0198900 0x7f35e0198d80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:28.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 -- 172.21.15.38:0/1718729372 >> 172.21.15.38:0/1718729372 conn(0x7f35e00fc1b0 msgr2=0x7f35e01021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:28.337 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 -- 172.21.15.38:0/1718729372 shutdown_connections 2024-03-26T06:29:28.338 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:28.335+0000 7f35e603d700 1 -- 172.21.15.38:0/1718729372 wait complete. 2024-03-26T06:29:28.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:28 smithi038 ceph-mon[32265]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T06:29:28.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:28 smithi038 ceph-mon[32265]: Reconfiguring daemon mon.a on smithi038 2024-03-26T06:29:28.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:28 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:28.861 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:28 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1718729372' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:28.862 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T06:29:28.862 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:29:28.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T06:29:28.892 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:29:28.892 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:29:28.964 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-26T06:29:28.964 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T06:29:28.999 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-26T06:29:29.000 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:29:29.070 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-26T06:29:29.070 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T06:29:29.103 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:28 smithi170 ceph-mon[36912]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T06:29:29.103 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:28 smithi170 ceph-mon[36912]: Reconfiguring daemon mon.a on smithi038 2024-03-26T06:29:29.103 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:28 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:29.103 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:28 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1718729372' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:29.105 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-26T06:29:29.106 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:29:29.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:28 smithi121 ceph-mon[36349]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T06:29:29.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:28 smithi121 ceph-mon[36349]: Reconfiguring daemon mon.a on smithi038 2024-03-26T06:29:29.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:28 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:29:29.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:28 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1718729372' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:29.176 INFO:tasks.cephadm:Adding mgr.x on smithi038 2024-03-26T06:29:29.177 INFO:tasks.cephadm:Adding mgr.y on smithi121 2024-03-26T06:29:29.177 INFO:tasks.cephadm:Adding mgr.z on smithi170 2024-03-26T06:29:29.177 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch apply mgr '3;smithi038=x;smithi121=y;smithi170=z' 2024-03-26T06:29:29.450 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:29:30.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:29 smithi121 ceph-mon[36349]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:30.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:30.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:30.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:30.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T06:29:30.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:30.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:29 smithi170 ceph-mon[36912]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:30.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:30.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:30.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:30.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T06:29:30.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:30.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:29 smithi038 ceph-mon[32265]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:30.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:30.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:30.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:30.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T06:29:30.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:30.896 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.891+0000 7fedc57a0700 1 -- 172.21.15.170:0/4005186498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedc0103020 msgr2=0x7fedc0103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:30.896 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.891+0000 7fedc57a0700 1 --2- 172.21.15.170:0/4005186498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedc0103020 0x7fedc0103440 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fedb0009910 tx=0x7fedb0009c20 comp rx=0 tx=0).stop 2024-03-26T06:29:30.896 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.891+0000 7fedc57a0700 1 -- 172.21.15.170:0/4005186498 shutdown_connections 2024-03-26T06:29:30.896 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.891+0000 7fedc57a0700 1 --2- 172.21.15.170:0/4005186498 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedc0103020 0x7fedc0103440 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:30.896 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.891+0000 7fedc57a0700 1 -- 172.21.15.170:0/4005186498 >> 172.21.15.170:0/4005186498 conn(0x7fedc00fcee0 msgr2=0x7fedc00ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:30.896 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.891+0000 7fedc57a0700 1 -- 172.21.15.170:0/4005186498 shutdown_connections 2024-03-26T06:29:30.896 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.891+0000 7fedc57a0700 1 -- 172.21.15.170:0/4005186498 wait complete. 2024-03-26T06:29:30.896 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.892+0000 7fedc57a0700 1 Processor -- start 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.892+0000 7fedc57a0700 1 -- start start 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.892+0000 7fedc57a0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedc0103020 0x7fedc0197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.892+0000 7fedc57a0700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedc01977b0 0x7fedc019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.892+0000 7fedc57a0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedc0197c30 0x7fedc019edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.892+0000 7fedc57a0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc0107a90 con 0x7fedc0197c30 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.892+0000 7fedc57a0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc0107910 con 0x7fedc0103020 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.892+0000 7fedc57a0700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedc0107c10 con 0x7fedc01977b0 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.892+0000 7fedb67fc700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedc01977b0 0x7fedc019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.892+0000 7fedb67fc700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedc01977b0 0x7fedc019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:60902/0 (socket says 172.21.15.170:60902) 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.892+0000 7fedb67fc700 1 -- 172.21.15.170:0/392563563 learned_addr learned my addr 172.21.15.170:0/392563563 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedbeffd700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedc0103020 0x7fedc0197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedbf7fe700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedc0197c30 0x7fedc019edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:30.897 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedb67fc700 1 -- 172.21.15.170:0/392563563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedc0103020 msgr2=0x7fedc0197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedb67fc700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedc0103020 0x7fedc0197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedb67fc700 1 -- 172.21.15.170:0/392563563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedc0197c30 msgr2=0x7fedc019edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedb67fc700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedc0197c30 0x7fedc019edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedb67fc700 1 -- 172.21.15.170:0/392563563 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc019f680 con 0x7fedc01977b0 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedbeffd700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedc0103020 0x7fedc0197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedb67fc700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedc01977b0 0x7fedc019c820 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feda800b780 tx=0x7feda800bb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedbcff9700 1 -- 172.21.15.170:0/392563563 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda8004230 con 0x7fedc01977b0 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedc57a0700 1 -- 172.21.15.170:0/392563563 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedc019f970 con 0x7fedc01977b0 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedbcff9700 1 -- 172.21.15.170:0/392563563 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feda80043d0 con 0x7fedc01977b0 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedbcff9700 1 -- 172.21.15.170:0/392563563 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda8011c60 con 0x7fedc01977b0 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.893+0000 7fedc57a0700 1 -- 172.21.15.170:0/392563563 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedc019ff80 con 0x7fedc01977b0 2024-03-26T06:29:30.898 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.894+0000 7fedbcff9700 1 -- 172.21.15.170:0/392563563 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7feda801a400 con 0x7fedc01977b0 2024-03-26T06:29:30.899 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.894+0000 7fedc57a0700 1 -- 172.21.15.170:0/392563563 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedc0063410 con 0x7fedc01977b0 2024-03-26T06:29:30.899 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.894+0000 7fedbcff9700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedac039450 0x7fedac03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:30.899 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.894+0000 7fedbcff9700 1 -- 172.21.15.170:0/392563563 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7feda804bd60 con 0x7fedc01977b0 2024-03-26T06:29:30.899 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.894+0000 7fedbeffd700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedac039450 0x7fedac03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:30.899 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.895+0000 7fedbeffd700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedac039450 0x7fedac03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fedb0005330 tx=0x7fedb0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:30.901 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:30.899+0000 7fedbcff9700 1 -- 172.21.15.170:0/392563563 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7feda8020500 con 0x7fedc01977b0 2024-03-26T06:29:31.106 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.105+0000 7fedc57a0700 1 -- 172.21.15.170:0/392563563 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi121=y;smithi170=z", "target": ["mon-mgr", ""]}) v1 -- 0x7fedc0062220 con 0x7fedac039450 2024-03-26T06:29:31.107 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:30 smithi170 ceph-mon[36912]: Reconfiguring mon.b (monmap changed)... 2024-03-26T06:29:31.107 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:30 smithi170 ceph-mon[36912]: Reconfiguring daemon mon.b on smithi121 2024-03-26T06:29:31.112 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:30 smithi121 ceph-mon[36349]: Reconfiguring mon.b (monmap changed)... 2024-03-26T06:29:31.112 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:30 smithi121 ceph-mon[36349]: Reconfiguring daemon mon.b on smithi121 2024-03-26T06:29:31.114 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mgr update... 2024-03-26T06:29:31.115 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.112+0000 7fedbcff9700 1 -- 172.21.15.170:0/392563563 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fedc0062220 con 0x7fedac039450 2024-03-26T06:29:31.116 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 -- 172.21.15.170:0/392563563 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedac039450 msgr2=0x7fedac03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:31.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedac039450 0x7fedac03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fedb0005330 tx=0x7fedb0005fb0 comp rx=0 tx=0).stop 2024-03-26T06:29:31.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 -- 172.21.15.170:0/392563563 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedc01977b0 msgr2=0x7fedc019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:31.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedc01977b0 0x7fedc019c820 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7feda800b780 tx=0x7feda800bb40 comp rx=0 tx=0).stop 2024-03-26T06:29:31.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 -- 172.21.15.170:0/392563563 shutdown_connections 2024-03-26T06:29:31.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fedc0197c30 0x7fedc019edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:31.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedac039450 0x7fedac03b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:31.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fedc0103020 0x7fedc0197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:31.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 --2- 172.21.15.170:0/392563563 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fedc01977b0 0x7fedc019c820 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:31.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 -- 172.21.15.170:0/392563563 >> 172.21.15.170:0/392563563 conn(0x7fedc00fcee0 msgr2=0x7fedc00fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:31.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 -- 172.21.15.170:0/392563563 shutdown_connections 2024-03-26T06:29:31.117 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:29:31.115+0000 7fedc57a0700 1 -- 172.21.15.170:0/392563563 wait complete. 2024-03-26T06:29:31.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:30 smithi038 ceph-mon[32265]: Reconfiguring mon.b (monmap changed)... 2024-03-26T06:29:31.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:30 smithi038 ceph-mon[32265]: Reconfiguring daemon mon.b on smithi121 2024-03-26T06:29:31.662 DEBUG:teuthology.orchestra.run.smithi121:mgr.y> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T06:29:31.665 DEBUG:teuthology.orchestra.run.smithi170:mgr.z> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.z.service 2024-03-26T06:29:31.666 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T06:29:31.667 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:29:31.667 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T06:29:31.686 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T06:29:31.686 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2024-03-26T06:29:31.744 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T06:29:31.744 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:31.744 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T06:29:31.744 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:31.744 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:31.744 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 06:28:36.708364369 +0000 2024-03-26T06:29:31.744 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 06:20:35.154934065 +0000 2024-03-26T06:29:31.744 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 06:20:35.154934065 +0000 2024-03-26T06:29:31.744 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 06:20:35.154934065 +0000 2024-03-26T06:29:31.745 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T06:29:31.814 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T06:29:31.814 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T06:29:31.814 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.00029291 s, 1.7 MB/s 2024-03-26T06:29:31.815 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T06:29:31.874 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2024-03-26T06:29:31.930 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T06:29:31.931 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:31.931 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T06:29:31.931 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:31.931 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:31.931 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 06:28:36.736363817 +0000 2024-03-26T06:29:31.931 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 06:20:35.495927491 +0000 2024-03-26T06:29:31.931 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 06:20:35.495927491 +0000 2024-03-26T06:29:31.931 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 06:20:35.495927491 +0000 2024-03-26T06:29:31.931 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T06:29:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T06:29:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T06:29:31.998 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000278324 s, 1.8 MB/s 2024-03-26T06:29:31.999 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T06:29:32.056 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2024-03-26T06:29:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T06:29:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:32.112 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-26T06:29:32.112 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:32.112 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:32.112 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 06:28:36.763363286 +0000 2024-03-26T06:29:32.112 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 06:20:35.851920628 +0000 2024-03-26T06:29:32.112 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 06:20:35.851920628 +0000 2024-03-26T06:29:32.112 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 06:20:35.851920628 +0000 2024-03-26T06:29:32.112 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T06:29:32.146 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:32 smithi170 ceph-mon[36912]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:32.146 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:32 smithi170 ceph-mon[36912]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi121=y;smithi170=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:29:32.146 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:32 smithi170 ceph-mon[36912]: Saving service mgr spec with placement smithi038=x;smithi121=y;smithi170=z;count:3 2024-03-26T06:29:32.146 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:32 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:32.147 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:32 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:32.147 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:32 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:32.147 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:32 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:32.147 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:32 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T06:29:32.147 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:32 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:32.178 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T06:29:32.178 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T06:29:32.178 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000284456 s, 1.8 MB/s 2024-03-26T06:29:32.179 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T06:29:32.235 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2024-03-26T06:29:32.290 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T06:29:32.291 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:32.291 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-26T06:29:32.291 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:32.291 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:32.291 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 06:28:36.794362675 +0000 2024-03-26T06:29:32.291 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 06:20:36.204913822 +0000 2024-03-26T06:29:32.291 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 06:20:36.204913822 +0000 2024-03-26T06:29:32.291 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 06:20:36.204913822 +0000 2024-03-26T06:29:32.291 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T06:29:32.356 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2024-03-26T06:29:32.356 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2024-03-26T06:29:32.356 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000283901 s, 1.8 MB/s 2024-03-26T06:29:32.358 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T06:29:32.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:32 smithi121 ceph-mon[36349]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:32.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:32 smithi121 ceph-mon[36349]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi121=y;smithi170=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:29:32.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:32 smithi121 ceph-mon[36349]: Saving service mgr spec with placement smithi038=x;smithi121=y;smithi170=z;count:3 2024-03-26T06:29:32.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:32 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:32.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:32 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:32.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:32 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:32.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:32 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:32.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:32 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T06:29:32.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:32 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:32.413 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-03-26T06:29:32.413 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T06:29:32.430 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T06:29:32.430 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2024-03-26T06:29:32.486 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T06:29:32.486 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:32.486 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-26T06:29:32.486 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:32.486 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:32.486 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-26 06:29:19.367326824 +0000 2024-03-26T06:29:32.486 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-26 06:20:44.785489377 +0000 2024-03-26T06:29:32.486 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-26 06:20:44.785489377 +0000 2024-03-26T06:29:32.486 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-26 06:20:44.785489377 +0000 2024-03-26T06:29:32.487 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T06:29:32.552 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-26T06:29:32.552 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-26T06:29:32.552 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.00029514 s, 1.7 MB/s 2024-03-26T06:29:32.553 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T06:29:32.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:32 smithi038 ceph-mon[32265]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:32.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:32 smithi038 ceph-mon[32265]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi121=y;smithi170=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:29:32.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:32 smithi038 ceph-mon[32265]: Saving service mgr spec with placement smithi038=x;smithi121=y;smithi170=z;count:3 2024-03-26T06:29:32.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:32 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:32.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:32 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:32.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:32 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:32.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:32 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T06:29:32.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:32 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T06:29:32.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:32 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:32.610 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2024-03-26T06:29:32.665 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T06:29:32.665 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:32.665 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-26T06:29:32.665 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:32.666 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:32.666 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-26 06:29:19.367326824 +0000 2024-03-26T06:29:32.666 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-26 06:20:45.130484704 +0000 2024-03-26T06:29:32.666 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-26 06:20:45.130484704 +0000 2024-03-26T06:29:32.666 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-26 06:20:45.130484704 +0000 2024-03-26T06:29:32.666 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T06:29:32.730 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-26T06:29:32.730 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-26T06:29:32.730 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000205366 s, 2.5 MB/s 2024-03-26T06:29:32.731 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T06:29:32.788 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2024-03-26T06:29:32.842 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T06:29:32.843 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:32.843 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-26T06:29:32.843 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:32.843 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:32.843 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-26 06:29:19.367326824 +0000 2024-03-26T06:29:32.843 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-26 06:20:45.476480018 +0000 2024-03-26T06:29:32.843 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-26 06:20:45.476480018 +0000 2024-03-26T06:29:32.843 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-26 06:20:45.476480018 +0000 2024-03-26T06:29:32.843 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T06:29:32.908 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-26T06:29:32.908 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-26T06:29:32.908 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000201199 s, 2.5 MB/s 2024-03-26T06:29:32.909 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T06:29:32.965 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2024-03-26T06:29:33.024 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T06:29:33.024 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:33.024 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-26T06:29:33.024 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:33.024 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:33.025 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2024-03-26 06:29:19.367326824 +0000 2024-03-26T06:29:33.025 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2024-03-26 06:20:45.831475209 +0000 2024-03-26T06:29:33.025 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2024-03-26 06:20:45.831475209 +0000 2024-03-26T06:29:33.025 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: 2024-03-26 06:20:45.831475209 +0000 2024-03-26T06:29:33.025 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T06:29:33.088 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2024-03-26T06:29:33.088 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2024-03-26T06:29:33.088 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.00018106 s, 2.8 MB/s 2024-03-26T06:29:33.089 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T06:29:33.146 DEBUG:teuthology.orchestra.run.smithi170:> set -ex 2024-03-26T06:29:33.146 DEBUG:teuthology.orchestra.run.smithi170:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T06:29:33.165 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T06:29:33.165 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_1 2024-03-26T06:29:33.223 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T06:29:33.223 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:33.223 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-26T06:29:33.223 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:33.223 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:33.223 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-26 06:29:09.736029638 +0000 2024-03-26T06:29:33.223 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-26 06:23:34.547988173 +0000 2024-03-26T06:29:33.223 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-26 06:23:34.547988173 +0000 2024-03-26T06:29:33.224 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-26 06:23:34.547988173 +0000 2024-03-26T06:29:33.224 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T06:29:33.293 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-26T06:29:33.293 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-26T06:29:33.294 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000104246 s, 4.9 MB/s 2024-03-26T06:29:33.295 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T06:29:33.352 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_2 2024-03-26T06:29:33.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:33 smithi121 ceph-mon[36349]: Reconfiguring mon.c (monmap changed)... 2024-03-26T06:29:33.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:33 smithi121 ceph-mon[36349]: Reconfiguring daemon mon.c on smithi170 2024-03-26T06:29:33.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:33.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:33.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:33.409 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T06:29:33.409 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:33.409 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-26T06:29:33.409 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:33.409 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:33.410 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-26 06:29:09.737029614 +0000 2024-03-26T06:29:33.410 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-26 06:23:34.880980235 +0000 2024-03-26T06:29:33.410 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-26 06:23:34.880980235 +0000 2024-03-26T06:29:33.410 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-26 06:23:34.880980235 +0000 2024-03-26T06:29:33.410 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T06:29:33.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:33 smithi170 ceph-mon[36912]: Reconfiguring mon.c (monmap changed)... 2024-03-26T06:29:33.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:33 smithi170 ceph-mon[36912]: Reconfiguring daemon mon.c on smithi170 2024-03-26T06:29:33.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:33.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:33.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:33.445 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-26T06:29:33.446 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-26T06:29:33.446 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 0.000118539 s, 4.3 MB/s 2024-03-26T06:29:33.448 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T06:29:33.504 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_3 2024-03-26T06:29:33.559 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T06:29:33.559 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:33.559 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-03-26T06:29:33.559 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:33.559 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:33.559 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-26 06:29:09.737029614 +0000 2024-03-26T06:29:33.559 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-26 06:23:35.229971914 +0000 2024-03-26T06:29:33.560 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-26 06:23:35.229971914 +0000 2024-03-26T06:29:33.560 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-26 06:23:35.229971914 +0000 2024-03-26T06:29:33.560 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T06:29:33.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:33 smithi038 ceph-mon[32265]: Reconfiguring mon.c (monmap changed)... 2024-03-26T06:29:33.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:33 smithi038 ceph-mon[32265]: Reconfiguring daemon mon.c on smithi170 2024-03-26T06:29:33.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:33.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:33.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:33.625 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-26T06:29:33.626 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-26T06:29:33.626 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.5283e-05 s, 5.4 MB/s 2024-03-26T06:29:33.627 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T06:29:33.683 DEBUG:teuthology.orchestra.run.smithi170:> stat /dev/vg_nvme/lv_4 2024-03-26T06:29:33.738 INFO:teuthology.orchestra.run.smithi170.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T06:29:33.738 INFO:teuthology.orchestra.run.smithi170.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T06:29:33.738 INFO:teuthology.orchestra.run.smithi170.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-26T06:29:33.738 INFO:teuthology.orchestra.run.smithi170.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:29:33.738 INFO:teuthology.orchestra.run.smithi170.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T06:29:33.739 INFO:teuthology.orchestra.run.smithi170.stdout:Access: 2024-03-26 06:29:09.737029614 +0000 2024-03-26T06:29:33.739 INFO:teuthology.orchestra.run.smithi170.stdout:Modify: 2024-03-26 06:23:35.575963667 +0000 2024-03-26T06:29:33.739 INFO:teuthology.orchestra.run.smithi170.stdout:Change: 2024-03-26 06:23:35.575963667 +0000 2024-03-26T06:29:33.739 INFO:teuthology.orchestra.run.smithi170.stdout: Birth: 2024-03-26 06:23:35.575963667 +0000 2024-03-26T06:29:33.739 DEBUG:teuthology.orchestra.run.smithi170:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T06:29:33.804 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records in 2024-03-26T06:29:33.805 INFO:teuthology.orchestra.run.smithi170.stderr:1+0 records out 2024-03-26T06:29:33.805 INFO:teuthology.orchestra.run.smithi170.stderr:512 bytes copied, 9.4258e-05 s, 5.4 MB/s 2024-03-26T06:29:33.806 DEBUG:teuthology.orchestra.run.smithi170:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T06:29:33.862 INFO:tasks.cephadm:Deploying osd.0 on smithi038 with /dev/vg_nvme/lv_4... 2024-03-26T06:29:33.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T06:29:34.099 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:29:34.220 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:34 smithi038 ceph-mon[32265]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:34.220 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:34 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:34.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:34 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:34.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:34 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:34.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:34 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T06:29:34.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:34 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T06:29:34.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:34 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T06:29:34.221 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:34 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:34.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:34 smithi121 ceph-mon[36349]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:34.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:34 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:34.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:34 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:34.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:34 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:34.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:34 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T06:29:34.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:34 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T06:29:34.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:34 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T06:29:34.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:34 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:34.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:34 smithi170 ceph-mon[36912]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:34.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:34 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:34.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:34 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:34.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:34 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:34.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:34 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T06:29:34.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:34 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T06:29:34.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:34 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T06:29:34.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:34 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:35.218 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:35 smithi170 ceph-mon[36912]: Deploying daemon mgr.z on smithi170 2024-03-26T06:29:35.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:35 smithi121 ceph-mon[36349]: Deploying daemon mgr.z on smithi170 2024-03-26T06:29:35.417 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:35 smithi038 ceph-mon[32265]: Deploying daemon mgr.z on smithi170 2024-03-26T06:29:35.611 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:35 smithi170 systemd[1]: Starting Ceph mgr.z for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:29:35.917 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:35 smithi170 podman[38758]: 2024-03-26 06:29:35.512593384 +0000 UTC m=+0.017474971 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:29:35.917 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:35 smithi170 podman[38758]: 2024-03-26 06:29:35.632693166 +0000 UTC m=+0.137574756 container create 5062b4749cc800cef91d7edf1996ece6b171ff2309b42538a05496317c827453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-26T06:29:35.917 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:35 smithi170 podman[38758]: 2024-03-26 06:29:35.826583155 +0000 UTC m=+0.331464741 container init 5062b4749cc800cef91d7edf1996ece6b171ff2309b42538a05496317c827453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True) 2024-03-26T06:29:35.917 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:35 smithi170 podman[38758]: 2024-03-26 06:29:35.829378402 +0000 UTC m=+0.334259993 container start 5062b4749cc800cef91d7edf1996ece6b171ff2309b42538a05496317c827453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T06:29:35.917 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:35 smithi170 ceph-mgr[38777]: -- 172.21.15.170:0/3550656601 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5616c75165a0 con 0x5616c74eb000 2024-03-26T06:29:35.990 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T06:29:35.990 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T06:29:35.990 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-03-26T06:29:35.990 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-03-26T06:29:35.990 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127249 s, 824 MB/s 2024-03-26T06:29:35.990 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-03-26T06:29:36.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:36 smithi038 ceph-mon[32265]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:36 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:36 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:36 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:36 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T06:29:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:36 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T06:29:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:36 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T06:29:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:36 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:36.371 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:35 smithi170 bash[38758]: 5062b4749cc800cef91d7edf1996ece6b171ff2309b42538a05496317c827453 2024-03-26T06:29:36.372 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:35 smithi170 systemd[1]: Started Ceph mgr.z for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:29:36.372 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:36 smithi170 ceph-mon[36912]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:36.372 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:36 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:36.372 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:36 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:36.372 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:36 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:36.372 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:36 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T06:29:36.372 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:36 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T06:29:36.372 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:36 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T06:29:36.373 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:36 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:36.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:36 smithi121 ceph-mon[36349]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:36.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:36 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:36.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:36 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:36.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:36 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:36 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T06:29:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:36 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T06:29:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:36 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T06:29:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:36 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:36.667 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:36 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:36.369+0000 7f3f8575d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T06:29:36.667 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:36 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:36.462+0000 7f3f8575d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T06:29:36.733 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi038:vg_nvme/lv_4 2024-03-26T06:29:36.972 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:29:37.386 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:37 smithi170 ceph-mon[36912]: Deploying daemon mgr.y on smithi121 2024-03-26T06:29:37.386 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:37.096+0000 7f3f8575d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T06:29:37.386 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:37.188+0000 7f3f8575d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T06:29:37.386 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:37.328+0000 7f3f8575d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T06:29:37.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:37 smithi121 ceph-mon[36349]: Deploying daemon mgr.y on smithi121 2024-03-26T06:29:37.415 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:37 smithi038 ceph-mon[32265]: Deploying daemon mgr.y on smithi121 2024-03-26T06:29:37.644 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:37.424+0000 7f3f8575d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T06:29:37.645 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:37.509+0000 7f3f8575d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T06:29:37.899 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:37.783+0000 7f3f8575d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T06:29:37.899 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:37.897+0000 7f3f8575d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T06:29:38.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:38 smithi170 ceph-mon[36912]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:38.167 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:37.985+0000 7f3f8575d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T06:29:38.167 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:38 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:38.082+0000 7f3f8575d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T06:29:38.294 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:38 smithi038 ceph-mon[32265]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:38.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:38 smithi121 ceph-mon[36349]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:38.605 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:38 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:38.262+0000 7f3f8575d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T06:29:38.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.649+0000 7f4968990700 1 -- 172.21.15.38:0/3926314089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4964102940 msgr2=0x7f4964104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:38.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.649+0000 7f4968990700 1 --2- 172.21.15.38:0/3926314089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4964102940 0x7f4964104d30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f495400b0a0 tx=0x7f495400b3b0 comp rx=0 tx=0).stop 2024-03-26T06:29:38.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.649+0000 7f4968990700 1 -- 172.21.15.38:0/3926314089 shutdown_connections 2024-03-26T06:29:38.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.649+0000 7f4968990700 1 --2- 172.21.15.38:0/3926314089 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4964107ba0 0x7f4964109f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:38.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.649+0000 7f4968990700 1 --2- 172.21.15.38:0/3926314089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4964102940 0x7f4964104d30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:38.651 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.649+0000 7f4968990700 1 --2- 172.21.15.38:0/3926314089 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4964105270 0x7f4964107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:38.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.649+0000 7f4968990700 1 -- 172.21.15.38:0/3926314089 >> 172.21.15.38:0/3926314089 conn(0x7f49640fc2b0 msgr2=0x7f49640fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:38.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.649+0000 7f4968990700 1 -- 172.21.15.38:0/3926314089 shutdown_connections 2024-03-26T06:29:38.652 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.649+0000 7f4968990700 1 -- 172.21.15.38:0/3926314089 wait complete. 2024-03-26T06:29:38.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4968990700 1 Processor -- start 2024-03-26T06:29:38.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4968990700 1 -- start start 2024-03-26T06:29:38.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4968990700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4964102940 0x7f49641022a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:38.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4968990700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4964105270 0x7f49641008f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:38.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4968990700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4964107ba0 0x7f4964100e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:38.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4968990700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496410d9c0 con 0x7f4964105270 2024-03-26T06:29:38.654 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4968990700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496410d840 con 0x7f4964102940 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4968990700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496410db40 con 0x7f4964107ba0 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4961d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4964105270 0x7f49641008f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4961d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4964105270 0x7f49641008f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:52306/0 (socket says 172.21.15.38:52306) 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f496259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4964102940 0x7f49641022a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4961d9b700 1 -- 172.21.15.38:0/4131035989 learned_addr learned my addr 172.21.15.38:0/4131035989 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4962d9d700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4964107ba0 0x7f4964100e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4961d9b700 1 -- 172.21.15.38:0/4131035989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4964107ba0 msgr2=0x7f4964100e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4961d9b700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4964107ba0 0x7f4964100e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4961d9b700 1 -- 172.21.15.38:0/4131035989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4964102940 msgr2=0x7f49641022a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4961d9b700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4964102940 0x7f49641022a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.650+0000 7f4961d9b700 1 -- 172.21.15.38:0/4131035989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49641016e0 con 0x7f4964105270 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.651+0000 7f4961d9b700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4964105270 0x7f49641008f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f494c00e9a0 tx=0x7f494c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:38.655 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.651+0000 7f495b7fe700 1 -- 172.21.15.38:0/4131035989 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c00c9c0 con 0x7f4964105270 2024-03-26T06:29:38.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.651+0000 7f495b7fe700 1 -- 172.21.15.38:0/4131035989 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f494c00cb60 con 0x7f4964105270 2024-03-26T06:29:38.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.651+0000 7f495b7fe700 1 -- 172.21.15.38:0/4131035989 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f494c018660 con 0x7f4964105270 2024-03-26T06:29:38.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.651+0000 7f4968990700 1 -- 172.21.15.38:0/4131035989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49641a21c0 con 0x7f4964105270 2024-03-26T06:29:38.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.651+0000 7f4968990700 1 -- 172.21.15.38:0/4131035989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49641a2680 con 0x7f4964105270 2024-03-26T06:29:38.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.652+0000 7f495b7fe700 1 -- 172.21.15.38:0/4131035989 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f494c018800 con 0x7f4964105270 2024-03-26T06:29:38.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.652+0000 7f495b7fe700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4950039200 0x7f495003b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:29:38.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.652+0000 7f4968990700 1 -- 172.21.15.38:0/4131035989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4964068aa0 con 0x7f4964105270 2024-03-26T06:29:38.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.652+0000 7f495b7fe700 1 -- 172.21.15.38:0/4131035989 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f494c04c310 con 0x7f4964105270 2024-03-26T06:29:38.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.652+0000 7f496259c700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4950039200 0x7f495003b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:29:38.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.652+0000 7f496259c700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4950039200 0x7f495003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4954007fa0 tx=0x7f49540060c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:29:38.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.657+0000 7f495b7fe700 1 -- 172.21.15.38:0/4131035989 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f494c00fc70 con 0x7f4964105270 2024-03-26T06:29:38.873 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:38.871+0000 7f4968990700 1 -- 172.21.15.38:0/4131035989 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4964062220 con 0x7f4950039200 2024-03-26T06:29:38.894 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:38 smithi121 systemd[1]: Started Ceph mgr.y for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:29:39.394 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:38 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:38.956+0000 7f0fa3118200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T06:29:39.394 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:39.053+0000 7f0fa3118200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T06:29:39.557 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:39.287+0000 7f3f8575d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T06:29:39.557 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:39.424+0000 7f3f8575d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T06:29:39.822 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:39.728+0000 7f0fa3118200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T06:29:39.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-mon[36349]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:39.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:39.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-mon[36349]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:29:39.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:29:39.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:29:39.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:39.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:39 smithi038 ceph-mon[32265]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:39 smithi038 ceph-mon[32265]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:29:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:29:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:29:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:39.855 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:39.765+0000 7f3f8575d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T06:29:39.856 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:39.853+0000 7f3f8575d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T06:29:39.856 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-mon[36912]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:39.856 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.856 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.856 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.856 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:39.856 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:39.856 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-mon[36912]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:29:39.856 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:29:39.856 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:29:39.857 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:40.144 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:39.821+0000 7f0fa3118200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T06:29:40.144 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:39.967+0000 7f0fa3118200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T06:29:40.144 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:40 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:40.070+0000 7f0fa3118200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T06:29:40.166 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:40 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:40.055+0000 7f3f8575d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:29:40.415 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:40 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:40.153+0000 7f0fa3118200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T06:29:40.415 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:40 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:40.413+0000 7f0fa3118200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T06:29:40.521 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:40 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:40.240+0000 7f3f8575d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T06:29:40.701 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:40 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:40.521+0000 7f0fa3118200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T06:29:40.701 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:40 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:40.606+0000 7f0fa3118200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T06:29:40.701 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:40 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:40.699+0000 7f0fa3118200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T06:29:40.781 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:40 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:40.519+0000 7f3f8575d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:29:40.781 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:40 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:40.779+0000 7f3f8575d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T06:29:41.015 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:40 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:40.873+0000 7f0fa3118200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T06:29:41.167 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:41.011+0000 7f3f8575d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T06:29:41.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:41.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:41.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T06:29:41.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T06:29:41.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:41.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:41.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:41.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T06:29:41.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T06:29:41.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:41.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:41.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:41.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:41.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T06:29:41.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T06:29:41.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:42.075 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:41 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:41.807+0000 7f3f8575d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T06:29:42.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-mon[36349]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:42.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-mon[36349]: Reconfiguring mgr.x (unknown last config time)... 2024-03-26T06:29:42.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-mon[36349]: Reconfiguring daemon mgr.x on smithi038 2024-03-26T06:29:42.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:42.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2089050288' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ec203cd-dc26-4133-83b3-de1d661ddd0d"}]: dispatch 2024-03-26T06:29:42.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2089050288' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ec203cd-dc26-4133-83b3-de1d661ddd0d"}]': finished 2024-03-26T06:29:42.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-mon[36349]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T06:29:42.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:29:42.145 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:41.833+0000 7f0fa3118200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T06:29:42.146 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:41 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:41.962+0000 7f0fa3118200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T06:29:42.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:42 smithi038 ceph-mon[32265]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:42.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:42 smithi038 ceph-mon[32265]: Reconfiguring mgr.x (unknown last config time)... 2024-03-26T06:29:42.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:42 smithi038 ceph-mon[32265]: Reconfiguring daemon mgr.x on smithi038 2024-03-26T06:29:42.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:42 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:42.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:42 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2089050288' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ec203cd-dc26-4133-83b3-de1d661ddd0d"}]: dispatch 2024-03-26T06:29:42.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:42 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2089050288' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ec203cd-dc26-4133-83b3-de1d661ddd0d"}]': finished 2024-03-26T06:29:42.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:42 smithi038 ceph-mon[32265]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T06:29:42.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:42 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:29:42.378 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:42 smithi170 ceph-mon[36912]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:42.379 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:42 smithi170 ceph-mon[36912]: Reconfiguring mgr.x (unknown last config time)... 2024-03-26T06:29:42.379 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:42 smithi170 ceph-mon[36912]: Reconfiguring daemon mgr.x on smithi038 2024-03-26T06:29:42.379 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:42 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:42.379 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:42 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2089050288' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ec203cd-dc26-4133-83b3-de1d661ddd0d"}]: dispatch 2024-03-26T06:29:42.379 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:42 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2089050288' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ec203cd-dc26-4133-83b3-de1d661ddd0d"}]': finished 2024-03-26T06:29:42.379 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:42 smithi170 ceph-mon[36912]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T06:29:42.380 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:42 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:29:42.577 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:42.299+0000 7f0fa3118200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T06:29:42.577 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:42.387+0000 7f0fa3118200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T06:29:42.577 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:42.576+0000 7f0fa3118200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:29:42.667 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:42 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:42.376+0000 7f3f8575d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T06:29:42.667 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:42 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:42.472+0000 7f3f8575d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T06:29:42.894 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:42 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:42.755+0000 7f0fa3118200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T06:29:43.167 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:29:42 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:29:42.688+0000 7f3f8575d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T06:29:43.291 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:43 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:43.034+0000 7f0fa3118200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:29:43.291 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:43 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:43.289+0000 7f0fa3118200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T06:29:43.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:43.478+0000 7f495b7fe700 1 -- 172.21.15.38:0/4131035989 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 14) v1 ==== 91698+0+0 (secure 0 0 0) 0x7f494c00cd10 con 0x7f4964105270 2024-03-26T06:29:43.644 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:43 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:43.524+0000 7f0fa3118200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T06:29:43.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:43 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2761859432' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:29:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:43 smithi121 ceph-mon[36349]: Standby manager daemon z started 2024-03-26T06:29:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:43 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T06:29:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:43 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:29:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:43 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T06:29:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:43 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:29:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:43 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2761859432' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:29:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:43 smithi038 ceph-mon[32265]: Standby manager daemon z started 2024-03-26T06:29:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:43 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T06:29:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:43 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:29:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:43 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T06:29:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:43 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:29:43.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:43 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2761859432' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:29:43.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:43 smithi170 ceph-mon[36912]: Standby manager daemon z started 2024-03-26T06:29:43.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:43 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T06:29:43.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:43 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:29:43.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:43 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T06:29:43.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:43 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.170:0/1531777080' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:29:44.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:44 smithi121 ceph-mon[36349]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:44.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:44 smithi121 ceph-mon[36349]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-26T06:29:44.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:44 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T06:29:44.645 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:44 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:44.308+0000 7f0fa3118200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T06:29:44.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:44 smithi038 ceph-mon[32265]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:44.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:44 smithi038 ceph-mon[32265]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-26T06:29:44.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:44 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T06:29:44.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:44 smithi170 ceph-mon[36912]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:44.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:44 smithi170 ceph-mon[36912]: mgrmap e14: x(active, since 77s), standbys: z 2024-03-26T06:29:44.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:44 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T06:29:45.144 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:44 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:44.871+0000 7f0fa3118200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T06:29:45.144 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:44 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:44.966+0000 7f0fa3118200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T06:29:45.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:45.485+0000 7f495b7fe700 1 -- 172.21.15.38:0/4131035989 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f494c0173d0 con 0x7f4964105270 2024-03-26T06:29:45.490 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:29:45 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:29:45.178+0000 7f0fa3118200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T06:29:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:45 smithi038 ceph-mon[32265]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:45 smithi038 ceph-mon[32265]: Standby manager daemon y started 2024-03-26T06:29:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:45 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T06:29:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:45 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:29:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:45 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T06:29:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:45 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:29:45.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:45 smithi121 ceph-mon[36349]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:45.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:45 smithi121 ceph-mon[36349]: Standby manager daemon y started 2024-03-26T06:29:45.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:45 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T06:29:45.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:45 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:29:45.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:45 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T06:29:45.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:45 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:29:45.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:45 smithi170 ceph-mon[36912]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:45.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:45 smithi170 ceph-mon[36912]: Standby manager daemon y started 2024-03-26T06:29:45.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:45 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T06:29:45.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:45 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:29:45.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:45 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T06:29:45.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:45 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.121:0/131509074' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:29:46.713 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:46 smithi038 ceph-mon[32265]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-26T06:29:46.713 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:46 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T06:29:46.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:46 smithi121 ceph-mon[36349]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-26T06:29:46.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:46 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T06:29:46.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:46 smithi170 ceph-mon[36912]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-03-26T06:29:46.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:46 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T06:29:47.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:47 smithi038 ceph-mon[32265]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:47.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:47 smithi121 ceph-mon[36349]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:47.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:47 smithi170 ceph-mon[36912]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:49.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:49.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:49.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:49.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:49.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:49.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:49.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:49.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:49.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:49.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:49.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:49.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:49.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:49.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:49.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:49.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:49.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:49.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:50.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:50 smithi121 ceph-mon[36349]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:50.410 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:50 smithi038 ceph-mon[32265]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:50.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:50 smithi170 ceph-mon[36912]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:51.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:51 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T06:29:51.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:51 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:51.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:51 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T06:29:51.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:51 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:51.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:51 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T06:29:51.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:51 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:52.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:52 smithi170 ceph-mon[36912]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:52.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:52 smithi170 ceph-mon[36912]: Deploying daemon osd.0 on smithi038 2024-03-26T06:29:52.420 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:52 smithi038 ceph-mon[32265]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:52.420 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:52 smithi038 ceph-mon[32265]: Deploying daemon osd.0 on smithi038 2024-03-26T06:29:52.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:52 smithi121 ceph-mon[36349]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:52.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:52 smithi121 ceph-mon[36349]: Deploying daemon osd.0 on smithi038 2024-03-26T06:29:53.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:53 smithi038 ceph-mon[32265]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:53.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:53 smithi121 ceph-mon[36349]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:53.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:53 smithi170 ceph-mon[36912]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:55.452 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:55 smithi038 ceph-mon[32265]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:55.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:55 smithi121 ceph-mon[36349]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:55.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:55 smithi170 ceph-mon[36912]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:56.945 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:56.946 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:56.946 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:57.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:57.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:57.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:57.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:57.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:57.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:58.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:57 smithi038 ceph-mon[32265]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:58.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:57 smithi121 ceph-mon[36349]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:58.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:57 smithi170 ceph-mon[36912]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:59.280 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 0 on host 'smithi038' 2024-03-26T06:29:59.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.278+0000 7f495b7fe700 1 -- 172.21.15.38:0/4131035989 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4964062220 con 0x7f4950039200 2024-03-26T06:29:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 -- 172.21.15.38:0/4131035989 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4950039200 msgr2=0x7f495003b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4950039200 0x7f495003b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4954007fa0 tx=0x7f49540060c0 comp rx=0 tx=0).stop 2024-03-26T06:29:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 -- 172.21.15.38:0/4131035989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4964105270 msgr2=0x7f49641008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:29:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4964105270 0x7f49641008f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f494c00e9a0 tx=0x7f494c00ecb0 comp rx=0 tx=0).stop 2024-03-26T06:29:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 -- 172.21.15.38:0/4131035989 shutdown_connections 2024-03-26T06:29:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4964107ba0 0x7f4964100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4950039200 0x7f495003b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4964102940 0x7f49641022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 --2- 172.21.15.38:0/4131035989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4964105270 0x7f49641008f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:29:59.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 -- 172.21.15.38:0/4131035989 >> 172.21.15.38:0/4131035989 conn(0x7f49640fc2b0 msgr2=0x7f4964103400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:29:59.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 -- 172.21.15.38:0/4131035989 shutdown_connections 2024-03-26T06:29:59.284 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:29:59.281+0000 7f4968990700 1 -- 172.21.15.38:0/4131035989 wait complete. 2024-03-26T06:29:59.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:59 smithi038 ceph-mon[32265]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:59.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:59 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:59 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:59 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:59.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:59 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:59.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:59 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:59 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:59 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:59.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:29:59 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:59 smithi121 ceph-mon[36349]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:59.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:59 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:59 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:59 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:59.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:59 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:59.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:59 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:59 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:59 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:59.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:29:59 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:59 smithi170 ceph-mon[36912]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:29:59.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:59 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:59 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:59 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:29:59.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:59 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:29:59.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:59 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:59 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:59 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:29:59.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:29:59 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:29:59.920 DEBUG:teuthology.orchestra.run.smithi038:osd.0> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T06:29:59.922 INFO:tasks.cephadm:Deploying osd.1 on smithi121 with /dev/vg_nvme/lv_4... 2024-03-26T06:29:59.923 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T06:30:00.169 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:30:00.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:00 smithi038 ceph-mon[32265]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T06:30:00.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:00 smithi038 ceph-mon[32265]: overall HEALTH_OK 2024-03-26T06:30:00.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:00 smithi121 ceph-mon[36349]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T06:30:00.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:00 smithi121 ceph-mon[36349]: overall HEALTH_OK 2024-03-26T06:30:00.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:00 smithi170 ceph-mon[36912]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T06:30:00.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:00 smithi170 ceph-mon[36912]: overall HEALTH_OK 2024-03-26T06:30:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T06:30:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T06:30:01.799 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-03-26T06:30:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-03-26T06:30:01.799 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127187 s, 824 MB/s 2024-03-26T06:30:01.799 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-03-26T06:30:01.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:01 smithi121 ceph-mon[36349]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:30:01.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:01 smithi121 ceph-mon[36349]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T06:30:01.822 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:01 smithi121 ceph-mon[36349]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T06:30:01.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:01 smithi121 ceph-mon[36349]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:30:01.823 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:01 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:01 smithi038 ceph-mon[32265]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:30:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:01 smithi038 ceph-mon[32265]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T06:30:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:01 smithi038 ceph-mon[32265]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T06:30:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:01 smithi038 ceph-mon[32265]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:30:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:01.833 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:30:01 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0[44309]: 2024-03-26T06:30:01.561+0000 7f911aa64700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T06:30:01.833 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:30:01 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0[44309]: 2024-03-26T06:30:01.568+0000 7f9117058700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:30:01.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:01 smithi170 ceph-mon[36912]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:30:01.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:01 smithi170 ceph-mon[36912]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T06:30:01.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:01 smithi170 ceph-mon[36912]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T06:30:01.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:01 smithi170 ceph-mon[36912]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:30:01.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:01 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:02.327 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_4 2024-03-26T06:30:02.562 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:30:02.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:02 smithi038 ceph-mon[32265]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:30:02.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:02 smithi038 ceph-mon[32265]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T06:30:02.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:02 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:02.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:02 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:02.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:02 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:02.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:02 smithi121 ceph-mon[36349]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:30:02.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:02 smithi121 ceph-mon[36349]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T06:30:02.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:02 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:02.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:02 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:02.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:02 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:02.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:02 smithi170 ceph-mon[36912]: from='osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:30:02.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:02 smithi170 ceph-mon[36912]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T06:30:02.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:02 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:02.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:02 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:02.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:02 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:03.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:30:03.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:30:03.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:30:03.624 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243] boot 2024-03-26T06:30:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T06:30:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi038 2024-03-26T06:30:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:30:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi038 to 17093M 2024-03-26T06:30:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:03.625 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:03.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:30:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:30:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:30:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243] boot 2024-03-26T06:30:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T06:30:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi038 2024-03-26T06:30:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:30:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi038 to 17093M 2024-03-26T06:30:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:03.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:03.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:03.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:03.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:30:03.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:30:03.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T06:30:03.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: osd.0 [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243] boot 2024-03-26T06:30:03.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T06:30:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:30:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi038 2024-03-26T06:30:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:30:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi038 to 17093M 2024-03-26T06:30:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:04.097 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.095+0000 7f95fe5d1700 1 -- 172.21.15.121:0/4077745845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95f8105230 msgr2=0x7f95f8107620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.095+0000 7f95fe5d1700 1 --2- 172.21.15.121:0/4077745845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95f8105230 0x7f95f8107620 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f95ec009990 tx=0x7f95ec009ca0 comp rx=0 tx=0).stop 2024-03-26T06:30:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.095+0000 7f95fe5d1700 1 -- 172.21.15.121:0/4077745845 shutdown_connections 2024-03-26T06:30:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.095+0000 7f95fe5d1700 1 --2- 172.21.15.121:0/4077745845 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95f8107b60 0x7f95f8109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.095+0000 7f95fe5d1700 1 --2- 172.21.15.121:0/4077745845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95f8102900 0x7f95f8104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.095+0000 7f95fe5d1700 1 --2- 172.21.15.121:0/4077745845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95f8105230 0x7f95f8107620 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.095+0000 7f95fe5d1700 1 -- 172.21.15.121:0/4077745845 >> 172.21.15.121:0/4077745845 conn(0x7f95f80fc2d0 msgr2=0x7f95f80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:30:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.095+0000 7f95fe5d1700 1 -- 172.21.15.121:0/4077745845 shutdown_connections 2024-03-26T06:30:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.095+0000 7f95fe5d1700 1 -- 172.21.15.121:0/4077745845 wait complete. 2024-03-26T06:30:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.095+0000 7f95fe5d1700 1 Processor -- start 2024-03-26T06:30:04.098 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95fe5d1700 1 -- start start 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95fe5d1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95f8102900 0x7f95f8102260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95fe5d1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95f8105230 0x7f95f81008b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95fe5d1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95f8107b60 0x7f95f8100df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95fe5d1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f810d9f0 con 0x7f95f8102900 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95fe5d1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f810d870 con 0x7f95f8105230 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95fe5d1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f810db70 con 0x7f95f8107b60 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95f77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95f8105230 0x7f95f81008b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95f77fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95f8105230 0x7f95f81008b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:41246/0 (socket says 172.21.15.121:41246) 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95f77fe700 1 -- 172.21.15.121:0/3761201154 learned_addr learned my addr 172.21.15.121:0/3761201154 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95f7fff700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95f8102900 0x7f95f8102260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95f77fe700 1 -- 172.21.15.121:0/3761201154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95f8107b60 msgr2=0x7f95f8100df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:04.101 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95fcb6e700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95f8107b60 0x7f95f8100df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95f77fe700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95f8107b60 0x7f95f8100df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95f77fe700 1 -- 172.21.15.121:0/3761201154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95f8102900 msgr2=0x7f95f8102260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95f77fe700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95f8102900 0x7f95f8102260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95f77fe700 1 -- 172.21.15.121:0/3761201154 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f81016a0 con 0x7f95f8105230 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95fcb6e700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95f8107b60 0x7f95f8100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.096+0000 7f95f77fe700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95f8105230 0x7f95f81008b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0098e0 tx=0x7f95ec0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.097+0000 7f95f57fa700 1 -- 172.21.15.121:0/3761201154 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec027070 con 0x7f95f8105230 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.097+0000 7f95f57fa700 1 -- 172.21.15.121:0/3761201154 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95ec004150 con 0x7f95f8105230 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.097+0000 7f95fe5d1700 1 -- 172.21.15.121:0/3761201154 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f81a20e0 con 0x7f95f8105230 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.097+0000 7f95f57fa700 1 -- 172.21.15.121:0/3761201154 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ec00f460 con 0x7f95f8105230 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.097+0000 7f95fe5d1700 1 -- 172.21.15.121:0/3761201154 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f81a2530 con 0x7f95f8105230 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.098+0000 7f95fe5d1700 1 -- 172.21.15.121:0/3761201154 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f8068aa0 con 0x7f95f8105230 2024-03-26T06:30:04.102 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.099+0000 7f95f57fa700 1 -- 172.21.15.121:0/3761201154 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f95ec02a070 con 0x7f95f8105230 2024-03-26T06:30:04.103 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.099+0000 7f95f57fa700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f95e00a2880 0x7f95e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.099+0000 7f95f57fa700 1 -- 172.21.15.121:0/3761201154 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f95ec0ce650 con 0x7f95f8105230 2024-03-26T06:30:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.100+0000 7f95f7fff700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f95e00a2880 0x7f95e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:04.104 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.100+0000 7f95f7fff700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f95e00a2880 0x7f95e00a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f95e8005950 tx=0x7f95e800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:30:04.105 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.103+0000 7f95f57fa700 1 -- 172.21.15.121:0/3761201154 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f95ec013070 con 0x7f95f8105230 2024-03-26T06:30:04.307 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:04.304+0000 7f95fe5d1700 1 -- 172.21.15.121:0/3761201154 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f95f8062220 con 0x7f95e00a2880 2024-03-26T06:30:05.038 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:04 smithi121 ceph-mon[36349]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T06:30:05.038 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:04 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:30:05.038 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:04 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:30:05.038 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:04 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:05.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:04 smithi038 ceph-mon[32265]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T06:30:05.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:30:05.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:30:05.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:05.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:04 smithi170 ceph-mon[36912]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T06:30:05.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:04 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:30:05.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:04 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:30:05.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:04 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:05.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:05 smithi121 ceph-mon[36349]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:05.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:05 smithi121 ceph-mon[36349]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:30:06.068 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:05 smithi170 ceph-mon[36912]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:06.068 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:05 smithi170 ceph-mon[36912]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:30:06.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:05 smithi038 ceph-mon[32265]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:06.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:05 smithi038 ceph-mon[32265]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:30:08.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:07 smithi038 ceph-mon[32265]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:08.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:07 smithi038 ceph-mon[32265]: from='client.? 172.21.15.121:0/3510662412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f"}]: dispatch 2024-03-26T06:30:08.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:07 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f"}]: dispatch 2024-03-26T06:30:08.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:07 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f"}]': finished 2024-03-26T06:30:08.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:07 smithi038 ceph-mon[32265]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T06:30:08.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:07 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:08.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:07 smithi038 ceph-mon[32265]: from='client.? 172.21.15.121:0/1350090746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:30:08.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:07 smithi121 ceph-mon[36349]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:08.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:07 smithi121 ceph-mon[36349]: from='client.? 172.21.15.121:0/3510662412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f"}]: dispatch 2024-03-26T06:30:08.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:07 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f"}]: dispatch 2024-03-26T06:30:08.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:07 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f"}]': finished 2024-03-26T06:30:08.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:07 smithi121 ceph-mon[36349]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T06:30:08.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:07 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:08.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:07 smithi121 ceph-mon[36349]: from='client.? 172.21.15.121:0/1350090746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:30:08.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:07 smithi170 ceph-mon[36912]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:08.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:07 smithi170 ceph-mon[36912]: from='client.? 172.21.15.121:0/3510662412' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f"}]: dispatch 2024-03-26T06:30:08.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:07 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f"}]: dispatch 2024-03-26T06:30:08.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:07 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f"}]': finished 2024-03-26T06:30:08.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:07 smithi170 ceph-mon[36912]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T06:30:08.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:07 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:08.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:07 smithi170 ceph-mon[36912]: from='client.? 172.21.15.121:0/1350090746' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:30:10.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:09 smithi038 ceph-mon[32265]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:10.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:09 smithi121 ceph-mon[36349]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:10.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:09 smithi170 ceph-mon[36912]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:11.992 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:11 smithi121 ceph-mon[36349]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:11 smithi038 ceph-mon[32265]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:12.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:11 smithi170 ceph-mon[36912]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:13.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:13 smithi121 ceph-mon[36349]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:14.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:13 smithi038 ceph-mon[32265]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:14.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:13 smithi170 ceph-mon[36912]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:14.996 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:14 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T06:30:14.996 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:14 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:15.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:14 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T06:30:15.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:14 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:15.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:14 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T06:30:15.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:14 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:16.000 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:15 smithi121 ceph-mon[36349]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:16.001 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:15 smithi121 ceph-mon[36349]: Deploying daemon osd.1 on smithi121 2024-03-26T06:30:16.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:15 smithi038 ceph-mon[32265]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:16.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:15 smithi038 ceph-mon[32265]: Deploying daemon osd.1 on smithi121 2024-03-26T06:30:16.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:15 smithi170 ceph-mon[36912]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:16.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:15 smithi170 ceph-mon[36912]: Deploying daemon osd.1 on smithi121 2024-03-26T06:30:17.954 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:17 smithi121 ceph-mon[36349]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:18.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:17 smithi038 ceph-mon[32265]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:18.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:17 smithi170 ceph-mon[36912]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:20.076 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:19 smithi121 ceph-mon[36349]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:20.076 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:20.076 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:20.076 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:19 smithi038 ceph-mon[32265]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:20.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:19 smithi170 ceph-mon[36912]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:20.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:20.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:20.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:21.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:21 smithi121 ceph-mon[36349]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:21.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:21 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:21.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:21 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:21.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:21 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:21.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:21 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:21.873 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:21 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:22.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:21 smithi038 ceph-mon[32265]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:21 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:21 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:21 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:21 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:21 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:22.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:21 smithi170 ceph-mon[36912]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:22.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:21 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:22.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:21 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:22.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:21 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:22.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:21 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:22.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:21 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:23.075 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.071+0000 7f95f57fa700 1 -- 172.21.15.121:0/3761201154 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f95f8062220 con 0x7f95e00a2880 2024-03-26T06:30:23.075 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 1 on host 'smithi121' 2024-03-26T06:30:23.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 -- 172.21.15.121:0/3761201154 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f95e00a2880 msgr2=0x7f95e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:23.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f95e00a2880 0x7f95e00a4d40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f95e8005950 tx=0x7f95e800b410 comp rx=0 tx=0).stop 2024-03-26T06:30:23.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 -- 172.21.15.121:0/3761201154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95f8105230 msgr2=0x7f95f81008b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:23.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95f8105230 0x7f95f81008b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f95ec0098e0 tx=0x7f95ec0057c0 comp rx=0 tx=0).stop 2024-03-26T06:30:23.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 -- 172.21.15.121:0/3761201154 shutdown_connections 2024-03-26T06:30:23.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f95f8107b60 0x7f95f8100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:23.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95f8102900 0x7f95f8102260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:23.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f95e00a2880 0x7f95e00a4d40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:23.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 --2- 172.21.15.121:0/3761201154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f95f8105230 0x7f95f81008b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:23.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 -- 172.21.15.121:0/3761201154 >> 172.21.15.121:0/3761201154 conn(0x7f95f80fc2d0 msgr2=0x7f95f8103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:30:23.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 -- 172.21.15.121:0/3761201154 shutdown_connections 2024-03-26T06:30:23.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:30:23.074+0000 7f95fe5d1700 1 -- 172.21.15.121:0/3761201154 wait complete. 2024-03-26T06:30:23.652 DEBUG:teuthology.orchestra.run.smithi121:osd.1> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T06:30:23.654 INFO:tasks.cephadm:Deploying osd.2 on smithi170 with /dev/vg_nvme/lv_4... 2024-03-26T06:30:23.655 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T06:30:23.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:23 smithi121 ceph-mon[36349]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:23.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:23.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:23.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:23.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:23 smithi121 ceph-mon[36349]: from='osd.1 [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T06:30:23.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:23 smithi121 ceph-mon[36349]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T06:30:23.898 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:30:24.020 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:23 smithi170 ceph-mon[36912]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:24.020 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:24.020 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:24.020 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:24.020 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:23 smithi170 ceph-mon[36912]: from='osd.1 [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T06:30:24.020 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:23 smithi170 ceph-mon[36912]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T06:30:24.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:23 smithi038 ceph-mon[32265]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:24.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:24.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:24.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:24.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:23 smithi038 ceph-mon[32265]: from='osd.1 [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T06:30:24.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:23 smithi038 ceph-mon[32265]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T06:30:25.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:25 smithi121 ceph-mon[36349]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T06:30:25.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:25 smithi121 ceph-mon[36349]: from='osd.1 [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:30:25.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:25 smithi121 ceph-mon[36349]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T06:30:25.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:25 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:25.131 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:25 smithi121 ceph-mon[36349]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:30:25.132 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:30:25 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1[40958]: 2024-03-26T06:30:25.037+0000 7f5dc1fc3700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T06:30:25.132 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:30:25 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1[40958]: 2024-03-26T06:30:25.041+0000 7f5dbe5b7700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:30:25.242 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:25 smithi170 ceph-mon[36912]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T06:30:25.242 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:25 smithi170 ceph-mon[36912]: from='osd.1 [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:30:25.242 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:25 smithi170 ceph-mon[36912]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T06:30:25.242 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:25.242 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:25 smithi170 ceph-mon[36912]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:30:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:25 smithi038 ceph-mon[32265]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T06:30:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:25 smithi038 ceph-mon[32265]: from='osd.1 [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:30:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:25 smithi038 ceph-mon[32265]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T06:30:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:25 smithi038 ceph-mon[32265]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:30:25.789 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T06:30:25.789 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T06:30:25.789 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-26T06:30:25.789 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-26T06:30:25.789 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00836615 s, 1.3 GB/s 2024-03-26T06:30:25.789 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 2024-03-26T06:30:25.789 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-26T06:30:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:26 smithi038 ceph-mon[32265]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:26 smithi038 ceph-mon[32265]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:30:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:26 smithi038 ceph-mon[32265]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T06:30:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:26.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:26 smithi170 ceph-mon[36912]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:26.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:26 smithi170 ceph-mon[36912]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:30:26.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:26 smithi170 ceph-mon[36912]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T06:30:26.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:26.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:26.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:26.367 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_4 2024-03-26T06:30:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:26 smithi121 ceph-mon[36349]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:26 smithi121 ceph-mon[36349]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:30:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:26 smithi121 ceph-mon[36349]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T06:30:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:26.610 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:30:27.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:30:27.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:30:27.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: osd.1 [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522] boot 2024-03-26T06:30:27.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T06:30:27.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:27.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:27.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:27.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:30:27.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:27.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:27.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:27.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:27.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:30:27.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:30:27.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: osd.1 [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522] boot 2024-03-26T06:30:27.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T06:30:27.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:27.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:27.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:27.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:30:27.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:27.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:27.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:27.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:27.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:30:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:30:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: osd.1 [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522] boot 2024-03-26T06:30:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T06:30:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:30:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:30:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:28.211 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.209+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3524885717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20e4102f70 msgr2=0x7f20e41033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:28.211 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.209+0000 7f20e8cb9700 1 --2- 172.21.15.170:0/3524885717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20e4102f70 0x7f20e41033f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f20d8009990 tx=0x7f20d8009ca0 comp rx=0 tx=0).stop 2024-03-26T06:30:28.211 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.209+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3524885717 shutdown_connections 2024-03-26T06:30:28.211 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.209+0000 7f20e8cb9700 1 --2- 172.21.15.170:0/3524885717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e4103930 0x7f20e410a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:28.211 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.209+0000 7f20e8cb9700 1 --2- 172.21.15.170:0/3524885717 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e4101e10 0x7f20e4102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:28.211 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.209+0000 7f20e8cb9700 1 --2- 172.21.15.170:0/3524885717 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20e4102f70 0x7f20e41033f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:28.211 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.209+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3524885717 >> 172.21.15.170:0/3524885717 conn(0x7f20e40fd3f0 msgr2=0x7f20e40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:30:28.211 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.209+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3524885717 shutdown_connections 2024-03-26T06:30:28.211 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.209+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3524885717 wait complete. 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e8cb9700 1 Processor -- start 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e8cb9700 1 -- start start 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e8cb9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e4101e10 0x7f20e4197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e8cb9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e4102f70 0x7f20e4198520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e8cb9700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20e4103930 0x7f20e419f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e8cb9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e410da00 con 0x7f20e4101e10 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e8cb9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e410d880 con 0x7f20e4102f70 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e8cb9700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e410db80 con 0x7f20e4103930 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e2d9d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20e4103930 0x7f20e419f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e2d9d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20e4103930 0x7f20e419f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:40170/0 (socket says 172.21.15.170:40170) 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e2d9d700 1 -- 172.21.15.170:0/3495329742 learned_addr learned my addr 172.21.15.170:0/3495329742 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-26T06:30:28.214 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e259c700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e4101e10 0x7f20e4197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e1d9b700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e4102f70 0x7f20e4198520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e2d9d700 1 -- 172.21.15.170:0/3495329742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e4102f70 msgr2=0x7f20e4198520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e2d9d700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e4102f70 0x7f20e4198520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e2d9d700 1 -- 172.21.15.170:0/3495329742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e4101e10 msgr2=0x7f20e4197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e2d9d700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e4101e10 0x7f20e4197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e2d9d700 1 -- 172.21.15.170:0/3495329742 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e419fc40 con 0x7f20e4103930 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.210+0000 7f20e2d9d700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20e4103930 0x7f20e419f5a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f20dc007d40 tx=0x7f20dc00a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.211+0000 7f20d37fe700 1 -- 172.21.15.170:0/3495329742 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc016020 con 0x7f20e4103930 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.211+0000 7f20d37fe700 1 -- 172.21.15.170:0/3495329742 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20dc00acb0 con 0x7f20e4103930 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.211+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3495329742 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e419ff30 con 0x7f20e4103930 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.211+0000 7f20d37fe700 1 -- 172.21.15.170:0/3495329742 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc0115e0 con 0x7f20e4103930 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.211+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3495329742 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e41a0460 con 0x7f20e4103930 2024-03-26T06:30:28.215 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.212+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3495329742 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e4068aa0 con 0x7f20e4103930 2024-03-26T06:30:28.216 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.213+0000 7f20d37fe700 1 -- 172.21.15.170:0/3495329742 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f20dc019020 con 0x7f20e4103930 2024-03-26T06:30:28.216 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.213+0000 7f20d37fe700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f20cc0a2880 0x7f20cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:28.216 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.213+0000 7f20d37fe700 1 -- 172.21.15.170:0/3495329742 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f20dc0cd000 con 0x7f20e4103930 2024-03-26T06:30:28.217 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.213+0000 7f20e259c700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f20cc0a2880 0x7f20cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:28.217 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.214+0000 7f20e259c700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f20cc0a2880 0x7f20cc0a4d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009a10 tx=0x7f20d4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:30:28.219 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.218+0000 7f20d37fe700 1 -- 172.21.15.170:0/3495329742 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f20dc020020 con 0x7f20e4103930 2024-03-26T06:30:28.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:28 smithi038 ceph-mon[32265]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:28.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:28 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi121 2024-03-26T06:30:28.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:28 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi121 to 17080M 2024-03-26T06:30:28.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:28 smithi038 ceph-mon[32265]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T06:30:28.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:28 smithi121 ceph-mon[36349]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:28 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi121 2024-03-26T06:30:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:28 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi121 to 17080M 2024-03-26T06:30:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:28 smithi121 ceph-mon[36349]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T06:30:28.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:28 smithi170 ceph-mon[36912]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T06:30:28.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:28 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi121 2024-03-26T06:30:28.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:28 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi121 to 17080M 2024-03-26T06:30:28.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:28 smithi170 ceph-mon[36912]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T06:30:28.421 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:28.419+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3495329742 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f20e4062220 con 0x7f20cc0a2880 2024-03-26T06:30:29.310 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:30:29.310 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:30:29.311 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:29.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:30:29.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:30:29.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:29.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:30:29.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:30:29.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:30.197 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:30 smithi170 ceph-mon[36912]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:30.197 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:30 smithi170 ceph-mon[36912]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:30:30.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:30 smithi038 ceph-mon[32265]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:30.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:30 smithi038 ceph-mon[32265]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:30:30.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:30 smithi121 ceph-mon[36349]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:30.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:30 smithi121 ceph-mon[36349]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:30:31.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:31 smithi121 ceph-mon[36349]: from='client.? 172.21.15.170:0/891073593' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ebac77-3d71-4e0a-b07a-6797adda9dd0"}]: dispatch 2024-03-26T06:30:31.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:31 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ebac77-3d71-4e0a-b07a-6797adda9dd0"}]: dispatch 2024-03-26T06:30:31.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:31 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09ebac77-3d71-4e0a-b07a-6797adda9dd0"}]': finished 2024-03-26T06:30:31.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:31 smithi121 ceph-mon[36349]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T06:30:31.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:31 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:31.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:31 smithi170 ceph-mon[36912]: from='client.? 172.21.15.170:0/891073593' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ebac77-3d71-4e0a-b07a-6797adda9dd0"}]: dispatch 2024-03-26T06:30:31.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:31 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ebac77-3d71-4e0a-b07a-6797adda9dd0"}]: dispatch 2024-03-26T06:30:31.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:31 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09ebac77-3d71-4e0a-b07a-6797adda9dd0"}]': finished 2024-03-26T06:30:31.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:31 smithi170 ceph-mon[36912]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T06:30:31.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:31 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:31.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:31 smithi038 ceph-mon[32265]: from='client.? 172.21.15.170:0/891073593' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ebac77-3d71-4e0a-b07a-6797adda9dd0"}]: dispatch 2024-03-26T06:30:31.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:31 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09ebac77-3d71-4e0a-b07a-6797adda9dd0"}]: dispatch 2024-03-26T06:30:31.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:31 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09ebac77-3d71-4e0a-b07a-6797adda9dd0"}]': finished 2024-03-26T06:30:31.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:31 smithi038 ceph-mon[32265]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T06:30:31.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:31 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:32.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:32 smithi121 ceph-mon[36349]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:32.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:32 smithi121 ceph-mon[36349]: from='client.? 172.21.15.170:0/2543804377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:30:32.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:32 smithi170 ceph-mon[36912]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:32.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:32 smithi170 ceph-mon[36912]: from='client.? 172.21.15.170:0/2543804377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:30:32.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:32 smithi038 ceph-mon[32265]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:32.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:32 smithi038 ceph-mon[32265]: from='client.? 172.21.15.170:0/2543804377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:30:34.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:34 smithi121 ceph-mon[36349]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:34.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:34 smithi170 ceph-mon[36912]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:34.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:34 smithi038 ceph-mon[32265]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:36 smithi038 ceph-mon[32265]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:36.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:36 smithi121 ceph-mon[36349]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:36.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:36 smithi170 ceph-mon[36912]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:38.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:38 smithi170 ceph-mon[36912]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:38.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:38 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T06:30:38.287 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:38 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:38.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:38 smithi121 ceph-mon[36349]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:38.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:38 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T06:30:38.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:38 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:38.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:38 smithi038 ceph-mon[32265]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:38.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:38 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T06:30:38.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:38 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:39.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:39 smithi170 ceph-mon[36912]: Deploying daemon osd.2 on smithi170 2024-03-26T06:30:39.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:39 smithi121 ceph-mon[36349]: Deploying daemon osd.2 on smithi170 2024-03-26T06:30:39.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:39 smithi038 ceph-mon[32265]: Deploying daemon osd.2 on smithi170 2024-03-26T06:30:40.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:40 smithi121 ceph-mon[36349]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:40.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:40 smithi170 ceph-mon[36912]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:40.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:40 smithi038 ceph-mon[32265]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:42.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:42 smithi121 ceph-mon[36349]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:42.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:42 smithi170 ceph-mon[36912]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:42.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:42 smithi038 ceph-mon[32265]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:44.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:44 smithi121 ceph-mon[36349]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:44.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:44 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:44.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:44 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:44.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:44 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:44.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:44 smithi170 ceph-mon[36912]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:44.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:44 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:44.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:44 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:44.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:44 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:44.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:44 smithi038 ceph-mon[32265]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:44.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:44 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:44.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:44 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:44.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:44 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:46.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:46 smithi121 ceph-mon[36349]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:46.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:46 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:46.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:46 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:46.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:46 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:46.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:46 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:46.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:46 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:46.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:46 smithi170 ceph-mon[36912]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:46.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:46 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:46.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:46 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:46.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:46 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:46.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:46 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:46.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:46 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:46.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:46 smithi038 ceph-mon[32265]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:46.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:46 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:46.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:46 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:46.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:46 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:46.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:46 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:46.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:46 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:46.838 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.836+0000 7f20d37fe700 1 -- 172.21.15.170:0/3495329742 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f20e4062220 con 0x7f20cc0a2880 2024-03-26T06:30:46.839 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 2 on host 'smithi170' 2024-03-26T06:30:46.841 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3495329742 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f20cc0a2880 msgr2=0x7f20cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:46.841 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f20cc0a2880 0x7f20cc0a4d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f20d4009a10 tx=0x7f20d4009190 comp rx=0 tx=0).stop 2024-03-26T06:30:46.841 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3495329742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20e4103930 msgr2=0x7f20e419f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:46.841 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20e4103930 0x7f20e419f5a0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f20dc007d40 tx=0x7f20dc00a340 comp rx=0 tx=0).stop 2024-03-26T06:30:46.841 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3495329742 shutdown_connections 2024-03-26T06:30:46.841 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20e4103930 0x7f20e419f5a0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:46.841 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20e4101e10 0x7f20e4197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:46.842 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f20cc0a2880 0x7f20cc0a4d40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:46.842 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 --2- 172.21.15.170:0/3495329742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20e4102f70 0x7f20e4198520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:46.842 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3495329742 >> 172.21.15.170:0/3495329742 conn(0x7f20e40fd3f0 msgr2=0x7f20e4106300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:30:46.842 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3495329742 shutdown_connections 2024-03-26T06:30:46.842 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:30:46.839+0000 7f20e8cb9700 1 -- 172.21.15.170:0/3495329742 wait complete. 2024-03-26T06:30:47.379 DEBUG:teuthology.orchestra.run.smithi170:osd.2> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T06:30:47.382 INFO:tasks.cephadm:Deploying osd.3 on smithi038 with /dev/vg_nvme/lv_3... 2024-03-26T06:30:47.382 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T06:30:47.632 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:30:48.045 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:47 smithi038 ceph-mon[32265]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:48.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:48.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:48.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:48.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:47 smithi038 ceph-mon[32265]: from='osd.2 [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T06:30:48.046 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:47 smithi038 ceph-mon[32265]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T06:30:48.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:47 smithi121 ceph-mon[36349]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:48.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:48.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:48.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:48.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:47 smithi121 ceph-mon[36349]: from='osd.2 [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T06:30:48.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:47 smithi121 ceph-mon[36349]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T06:30:48.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:47 smithi170 ceph-mon[36912]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:48.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:48.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:48.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:30:48.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:47 smithi170 ceph-mon[36912]: from='osd.2 [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T06:30:48.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:47 smithi170 ceph-mon[36912]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T06:30:49.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:48 smithi038 ceph-mon[32265]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T06:30:49.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:48 smithi038 ceph-mon[32265]: from='osd.2 [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:30:49.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:48 smithi038 ceph-mon[32265]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T06:30:49.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:48 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:49.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:48 smithi038 ceph-mon[32265]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:30:49.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:48 smithi121 ceph-mon[36349]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T06:30:49.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:48 smithi121 ceph-mon[36349]: from='osd.2 [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:30:49.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:48 smithi121 ceph-mon[36349]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T06:30:49.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:48 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:49.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:48 smithi121 ceph-mon[36349]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:30:49.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:48 smithi170 ceph-mon[36912]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T06:30:49.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:48 smithi170 ceph-mon[36912]: from='osd.2 [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:30:49.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:48 smithi170 ceph-mon[36912]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T06:30:49.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:48 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:49.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:48 smithi170 ceph-mon[36912]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:30:49.603 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:30:49 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2[42088]: 2024-03-26T06:30:49.560+0000 7f4a1c337700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T06:30:49.603 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:30:49 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2[42088]: 2024-03-26T06:30:49.565+0000 7f4a14923700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:30:49.711 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T06:30:49.711 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T06:30:49.711 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-03-26T06:30:49.711 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-03-26T06:30:49.711 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0145048 s, 723 MB/s 2024-03-26T06:30:49.711 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-03-26T06:30:49.899 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:49 smithi170 ceph-mon[36912]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:49.899 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:49 smithi170 ceph-mon[36912]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:30:49.899 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:49 smithi170 ceph-mon[36912]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T06:30:49.900 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:49.900 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:49.900 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:49 smithi170 ceph-mon[36912]: from='osd.2 ' entity='osd.2' 2024-03-26T06:30:50.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:49 smithi121 ceph-mon[36349]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:50.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:49 smithi121 ceph-mon[36349]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:30:50.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:49 smithi121 ceph-mon[36349]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T06:30:50.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:50.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:50.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:49 smithi121 ceph-mon[36349]: from='osd.2 ' entity='osd.2' 2024-03-26T06:30:50.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:49 smithi038 ceph-mon[32265]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:50.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:49 smithi038 ceph-mon[32265]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:30:50.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:49 smithi038 ceph-mon[32265]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T06:30:50.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:50.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:50.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:49 smithi038 ceph-mon[32265]: from='osd.2 ' entity='osd.2' 2024-03-26T06:30:50.485 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi038:vg_nvme/lv_3 2024-03-26T06:30:50.731 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:30:51.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:30:51.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:30:51.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:51.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:30:51.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:51.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:51.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: osd.2 [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871] boot 2024-03-26T06:30:51.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T06:30:51.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:50 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:51.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:30:51.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:30:51.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:51.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:30:51.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:51.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:51.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: osd.2 [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871] boot 2024-03-26T06:30:51.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T06:30:51.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:50 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:51.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:30:51.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:30:51.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:51.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:30:51.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:51.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:30:51.178 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:51.179 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: osd.2 [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871] boot 2024-03-26T06:30:51.179 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T06:30:51.179 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:50 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:30:52.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:51 smithi121 ceph-mon[36349]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:52.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:51 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi170 2024-03-26T06:30:52.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:51 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi170 to 17080M 2024-03-26T06:30:52.151 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:51 smithi038 ceph-mon[32265]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:52.151 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:51 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi170 2024-03-26T06:30:52.151 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:51 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi170 to 17080M 2024-03-26T06:30:52.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:51 smithi170 ceph-mon[36912]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T06:30:52.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:51 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi170 2024-03-26T06:30:52.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:51 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi170 to 17080M 2024-03-26T06:30:52.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.374+0000 7f8da3492700 1 -- 172.21.15.38:0/688084417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d9c100d10 msgr2=0x7f8d9c101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:52.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.374+0000 7f8da3492700 1 --2- 172.21.15.38:0/688084417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d9c100d10 0x7f8d9c101130 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c0098f0 tx=0x7f8d8c009c00 comp rx=0 tx=0).stop 2024-03-26T06:30:52.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 -- 172.21.15.38:0/688084417 shutdown_connections 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 --2- 172.21.15.38:0/688084417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d9c1028b0 0x7f8d9c109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 --2- 172.21.15.38:0/688084417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d9c100d10 0x7f8d9c101130 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 --2- 172.21.15.38:0/688084417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d9c101f10 0x7f8d9c102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 -- 172.21.15.38:0/688084417 >> 172.21.15.38:0/688084417 conn(0x7f8d9c0fc2b0 msgr2=0x7f8d9c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 -- 172.21.15.38:0/688084417 shutdown_connections 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 -- 172.21.15.38:0/688084417 wait complete. 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 Processor -- start 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 -- start start 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d9c100d10 0x7f8d9c06a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d9c101f10 0x7f8d9c19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d9c1028b0 0x7f8d9c19f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:52.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c10c8d0 con 0x7f8d9c1028b0 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c10c750 con 0x7f8d9c101f10 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.375+0000 7f8da3492700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c10ca50 con 0x7f8d9c100d10 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da122e700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d9c100d10 0x7f8d9c06a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da1a2f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d9c1028b0 0x7f8d9c19f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da122e700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d9c100d10 0x7f8d9c06a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:44184/0 (socket says 172.21.15.38:44184) 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da122e700 1 -- 172.21.15.38:0/2776713663 learned_addr learned my addr 172.21.15.38:0/2776713663 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da0a2d700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d9c101f10 0x7f8d9c19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da122e700 1 -- 172.21.15.38:0/2776713663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d9c101f10 msgr2=0x7f8d9c19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da122e700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d9c101f10 0x7f8d9c19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da122e700 1 -- 172.21.15.38:0/2776713663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d9c1028b0 msgr2=0x7f8d9c19f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da122e700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d9c1028b0 0x7f8d9c19f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da122e700 1 -- 172.21.15.38:0/2776713663 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c19fd50 con 0x7f8d9c100d10 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da1a2f700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d9c1028b0 0x7f8d9c19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:30:52.380 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da122e700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d9c100d10 0x7f8d9c06a600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c000c00 tx=0x7f8d8c0050c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:30:52.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8d927fc700 1 -- 172.21.15.38:0/2776713663 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c005570 con 0x7f8d9c100d10 2024-03-26T06:30:52.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8d927fc700 1 -- 172.21.15.38:0/2776713663 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d8c00ba40 con 0x7f8d9c100d10 2024-03-26T06:30:52.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.376+0000 7f8da3492700 1 -- 172.21.15.38:0/2776713663 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9c19ffe0 con 0x7f8d9c100d10 2024-03-26T06:30:52.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.377+0000 7f8d927fc700 1 -- 172.21.15.38:0/2776713663 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c017400 con 0x7f8d9c100d10 2024-03-26T06:30:52.381 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.377+0000 7f8da3492700 1 -- 172.21.15.38:0/2776713663 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9c1a0380 con 0x7f8d9c100d10 2024-03-26T06:30:52.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.377+0000 7f8da3492700 1 -- 172.21.15.38:0/2776713663 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9c02d830 con 0x7f8d9c100d10 2024-03-26T06:30:52.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.379+0000 7f8d927fc700 1 -- 172.21.15.38:0/2776713663 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f8d8c00b560 con 0x7f8d9c100d10 2024-03-26T06:30:52.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.379+0000 7f8d927fc700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8d880a2800 0x7f8d880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:30:52.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.379+0000 7f8da0a2d700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8d880a2800 0x7f8d880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:30:52.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.379+0000 7f8d927fc700 1 -- 172.21.15.38:0/2776713663 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 2851+0+0 (secure 0 0 0) 0x7f8d8c0ce3a0 con 0x7f8d9c100d10 2024-03-26T06:30:52.382 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.380+0000 7f8da0a2d700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8d880a2800 0x7f8d880a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009750 tx=0x7f8d98009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:30:52.385 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.383+0000 7f8d927fc700 1 -- 172.21.15.38:0/2776713663 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8d8c05c260 con 0x7f8d9c100d10 2024-03-26T06:30:52.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:52.598+0000 7f8da3492700 1 -- 172.21.15.38:0/2776713663 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d9c062220 con 0x7f8d880a2800 2024-03-26T06:30:53.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:52 smithi121 ceph-mon[36349]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T06:30:53.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T06:30:53.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:30:53.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:30:53.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:53.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:52 smithi170 ceph-mon[36912]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T06:30:53.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T06:30:53.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:30:53.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:30:53.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:53.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:52 smithi038 ceph-mon[32265]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T06:30:53.216 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T06:30:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:30:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:30:53.217 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:30:54.134 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:53 smithi038 ceph-mon[32265]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:30:54.134 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:53 smithi038 ceph-mon[32265]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:30:54.134 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:53 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T06:30:54.134 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:53 smithi038 ceph-mon[32265]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T06:30:54.134 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:53 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T06:30:54.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:53 smithi121 ceph-mon[36349]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:30:54.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:53 smithi121 ceph-mon[36349]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:30:54.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:53 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T06:30:54.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:53 smithi121 ceph-mon[36349]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T06:30:54.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:53 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T06:30:54.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:53 smithi170 ceph-mon[36912]: pgmap v82: 0 pgs: ; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:30:54.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:53 smithi170 ceph-mon[36912]: from='client.24181 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:30:54.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:53 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T06:30:54.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:53 smithi170 ceph-mon[36912]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T06:30:54.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:53 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T06:30:54.410 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:30:54 smithi038 sudo[48909]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T06:30:54.410 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:30:54 smithi038 sudo[48909]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:54.410 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:30:54 smithi038 sudo[48909]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:54.411 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:30:54 smithi038 sudo[48909]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:54.833 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:30:54 smithi038 sudo[48917]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T06:30:54.833 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:30:54 smithi038 sudo[48917]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:54.833 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:30:54 smithi038 sudo[48917]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:54.833 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:30:54 smithi038 sudo[48917]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:55.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:54 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T06:30:55.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:54 smithi121 ceph-mon[36349]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T06:30:55.018 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:54 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:55.019 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:30:54 smithi121 sudo[44681]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T06:30:55.019 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:30:54 smithi121 sudo[44681]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:55.019 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:30:54 smithi121 sudo[44681]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:55.019 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:30:54 smithi121 sudo[44681]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:55.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:54 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T06:30:55.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:54 smithi170 ceph-mon[36912]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T06:30:55.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:54 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:55.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:54 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T06:30:55.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:54 smithi038 ceph-mon[32265]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T06:30:55.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:54 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:55.394 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:30:55 smithi121 sudo[44684]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T06:30:55.394 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:30:55 smithi121 sudo[44684]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:55.394 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:30:55 smithi121 sudo[44684]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:55.395 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:30:55 smithi121 sudo[44684]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:55.537 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:30:55 smithi170 sudo[46029]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T06:30:55.538 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:30:55 smithi170 sudo[46029]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:55.538 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:30:55 smithi170 sudo[46029]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:55.538 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:30:55 smithi170 sudo[46029]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:55.917 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:30:55 smithi170 sudo[46032]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T06:30:55.917 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:30:55 smithi170 sudo[46032]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:55.917 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:30:55 smithi170 sudo[46032]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:55.917 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:30:55 smithi170 sudo[46032]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:56.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:55 smithi038 sudo[48999]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T06:30:56.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:55 smithi038 sudo[48999]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:56.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:55 smithi038 sudo[48999]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:56.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:55 smithi038 sudo[48999]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 sudo[49002]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T06:30:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 sudo[49002]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 sudo[49002]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 sudo[49002]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 ceph-mon[32265]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:30:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 ceph-mon[32265]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T06:30:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/170228200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05893f6a-6566-412d-8341-72c9d577310b"}]: dispatch 2024-03-26T06:30:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/170228200' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05893f6a-6566-412d-8341-72c9d577310b"}]': finished 2024-03-26T06:30:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 ceph-mon[32265]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T06:30:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:30:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:56 smithi038 ceph-mon[32265]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T06:30:56.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 ceph-mon[36912]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:30:56.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 ceph-mon[36912]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T06:30:56.340 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:56.341 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/170228200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05893f6a-6566-412d-8341-72c9d577310b"}]: dispatch 2024-03-26T06:30:56.341 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/170228200' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05893f6a-6566-412d-8341-72c9d577310b"}]': finished 2024-03-26T06:30:56.341 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 ceph-mon[36912]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T06:30:56.341 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:30:56.341 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:56.341 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 ceph-mon[36912]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T06:30:56.341 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 sudo[46035]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T06:30:56.341 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 sudo[46035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:56.341 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 sudo[46035]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:56.341 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 sudo[46035]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:56.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 ceph-mon[36349]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:30:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 ceph-mon[36349]: osdmap e22: 3 total, 3 up, 3 in 2024-03-26T06:30:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/170228200' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "05893f6a-6566-412d-8341-72c9d577310b"}]: dispatch 2024-03-26T06:30:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/170228200' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "05893f6a-6566-412d-8341-72c9d577310b"}]': finished 2024-03-26T06:30:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 ceph-mon[36349]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T06:30:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:30:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:30:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 ceph-mon[36349]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T06:30:56.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:30:56.632+0000 7f8d927fc700 1 -- 172.21.15.38:0/2776713663 <== mon.1 v2:172.21.15.170:3300/0 7 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8d8c098a70 con 0x7f8d9c100d10 2024-03-26T06:30:56.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 sudo[46038]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T06:30:56.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 sudo[46038]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:56.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 sudo[46038]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:56.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:56 smithi170 sudo[46038]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:57.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 sudo[44687]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T06:30:57.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 sudo[44687]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:57.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 sudo[44687]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:57.073 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:56 smithi121 sudo[44687]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:57.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 sudo[44690]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T06:30:57.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 sudo[44690]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T06:30:57.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 sudo[44690]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T06:30:57.367 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 sudo[44690]: pam_unix(sudo:session): session closed for user root 2024-03-26T06:30:57.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T06:30:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:30:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:30:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:30:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T06:30:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:30:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:30:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:30:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/622236497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:30:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T06:30:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T06:30:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:30:57.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:30:57.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:30:57.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-26T06:30:57.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:57 smithi121 ceph-mon[36349]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T06:30:57.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T06:30:57.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:30:57.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:30:57.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/622236497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-26T06:30:57.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:57 smithi170 ceph-mon[36912]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T06:30:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T06:30:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:30:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:30:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:30:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T06:30:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:30:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:30:57.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:30:57.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/622236497' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:30:57.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T06:30:57.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T06:30:57.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:30:57.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:30:57.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:30:57.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-26T06:30:57.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:57 smithi038 ceph-mon[32265]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T06:30:58.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:58 smithi121 ceph-mon[36349]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:30:58.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:58 smithi170 ceph-mon[36912]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:30:58.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:58 smithi038 ceph-mon[32265]: pgmap v88: 1 pgs: 1 unknown; 0 B data, 93 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:30:59.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:30:59 smithi121 ceph-mon[36349]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:30:59.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:30:59 smithi170 ceph-mon[36912]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:30:59.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:30:59 smithi038 ceph-mon[32265]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:31:01.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:01 smithi038 ceph-mon[32265]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:31:01.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:01 smithi121 ceph-mon[36349]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:31:01.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:01 smithi170 ceph-mon[36912]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:31:03.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:03 smithi038 ceph-mon[32265]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-03-26T06:31:03.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:03 smithi121 ceph-mon[36349]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-03-26T06:31:03.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:03 smithi170 ceph-mon[36912]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 65 KiB/s wr, 6 op/s 2024-03-26T06:31:04.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:04 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T06:31:04.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:04 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T06:31:04.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:04.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:04 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T06:31:04.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:04 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:05.516 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:05 smithi038 ceph-mon[32265]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-26T06:31:05.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:05 smithi038 ceph-mon[32265]: Deploying daemon osd.3 on smithi038 2024-03-26T06:31:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:05 smithi121 ceph-mon[36349]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-26T06:31:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:05 smithi121 ceph-mon[36349]: Deploying daemon osd.3 on smithi038 2024-03-26T06:31:05.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:05 smithi170 ceph-mon[36912]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 44 KiB/s wr, 4 op/s 2024-03-26T06:31:05.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:05 smithi170 ceph-mon[36912]: Deploying daemon osd.3 on smithi038 2024-03-26T06:31:07.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:07 smithi170 ceph-mon[36912]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-26T06:31:07.720 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:07 smithi121 ceph-mon[36349]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-26T06:31:07.760 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:07 smithi038 ceph-mon[32265]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 33 KiB/s wr, 3 op/s 2024-03-26T06:31:09.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:09 smithi038 ceph-mon[32265]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T06:31:09.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:09 smithi170 ceph-mon[36912]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T06:31:09.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:09 smithi121 ceph-mon[36349]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T06:31:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:10 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:10 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:10 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:10.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:10 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:10.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:10 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:10.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:10 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:10.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:10 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:10.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:10 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:10.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:10 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:11.639 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:11 smithi038 ceph-mon[32265]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-26T06:31:11.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:11 smithi121 ceph-mon[36349]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-26T06:31:11.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:11 smithi170 ceph-mon[36912]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-26T06:31:13.059 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.057+0000 7f8d927fc700 1 -- 172.21.15.38:0/2776713663 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8d9c062220 con 0x7f8d880a2800 2024-03-26T06:31:13.059 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 3 on host 'smithi038' 2024-03-26T06:31:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.059+0000 7f8da3492700 1 -- 172.21.15.38:0/2776713663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8d880a2800 msgr2=0x7f8d880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:31:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.059+0000 7f8da3492700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8d880a2800 0x7f8d880a4cc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009750 tx=0x7f8d98009310 comp rx=0 tx=0).stop 2024-03-26T06:31:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.059+0000 7f8da3492700 1 -- 172.21.15.38:0/2776713663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d9c100d10 msgr2=0x7f8d9c06a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:31:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.059+0000 7f8da3492700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d9c100d10 0x7f8d9c06a600 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c000c00 tx=0x7f8d8c0050c0 comp rx=0 tx=0).stop 2024-03-26T06:31:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.060+0000 7f8da3492700 1 -- 172.21.15.38:0/2776713663 shutdown_connections 2024-03-26T06:31:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.060+0000 7f8da3492700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d9c1028b0 0x7f8d9c19f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.060+0000 7f8da3492700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8d9c100d10 0x7f8d9c06a600 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.060+0000 7f8da3492700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8d880a2800 0x7f8d880a4cc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.060+0000 7f8da3492700 1 --2- 172.21.15.38:0/2776713663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8d9c101f10 0x7f8d9c19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.060+0000 7f8da3492700 1 -- 172.21.15.38:0/2776713663 >> 172.21.15.38:0/2776713663 conn(0x7f8d9c0fc2b0 msgr2=0x7f8d9c105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:31:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.060+0000 7f8da3492700 1 -- 172.21.15.38:0/2776713663 shutdown_connections 2024-03-26T06:31:13.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:31:13.060+0000 7f8da3492700 1 -- 172.21.15.38:0/2776713663 wait complete. 2024-03-26T06:31:13.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:13.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:13.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:13.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:13.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:13.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:13.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:13.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:13.779 DEBUG:teuthology.orchestra.run.smithi038:osd.3> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T06:31:13.781 INFO:tasks.cephadm:Deploying osd.4 on smithi121 with /dev/vg_nvme/lv_3... 2024-03-26T06:31:13.782 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T06:31:14.023 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:31:14.332 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:14 smithi038 ceph-mon[32265]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-26T06:31:14.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:14 smithi038 ceph-mon[32265]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T06:31:14.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:14 smithi170 ceph-mon[36912]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-26T06:31:14.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:14 smithi170 ceph-mon[36912]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T06:31:14.456 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:14 smithi121 ceph-mon[36349]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 853 B/s rd, 22 KiB/s wr, 2 op/s 2024-03-26T06:31:14.456 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:14 smithi121 ceph-mon[36349]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T06:31:15.391 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:15 smithi121 ceph-mon[36349]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T06:31:15.391 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:15 smithi121 ceph-mon[36349]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T06:31:15.391 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:15 smithi121 ceph-mon[36349]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:31:15.391 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:15 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:15.391 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:15 smithi121 ceph-mon[36349]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:31:15.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:15 smithi038 ceph-mon[32265]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T06:31:15.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:15 smithi038 ceph-mon[32265]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T06:31:15.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:15 smithi038 ceph-mon[32265]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:31:15.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:15.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:15 smithi038 ceph-mon[32265]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:31:15.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:15 smithi170 ceph-mon[36912]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T06:31:15.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:15 smithi170 ceph-mon[36912]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T06:31:15.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:15 smithi170 ceph-mon[36912]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:31:15.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:15.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:15 smithi170 ceph-mon[36912]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:31:15.947 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T06:31:15.948 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T06:31:15.948 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-03-26T06:31:15.948 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-03-26T06:31:15.948 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112096 s, 935 MB/s 2024-03-26T06:31:15.948 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-03-26T06:31:16.282 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:16 smithi121 ceph-mon[36349]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:31:16.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:16 smithi121 ceph-mon[36349]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T06:31:16.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:16 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:16.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:16 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:16.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:16 smithi121 ceph-mon[36349]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' 2024-03-26T06:31:16.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:16 smithi038 ceph-mon[32265]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:31:16.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:16 smithi038 ceph-mon[32265]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T06:31:16.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:16 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:16.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:16 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:16.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:16 smithi038 ceph-mon[32265]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' 2024-03-26T06:31:16.334 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:31:16 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3[51003]: 2024-03-26T06:31:16.020+0000 7f98d59b8700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T06:31:16.334 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:31:16 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3[51003]: 2024-03-26T06:31:16.030+0000 7f98cdfa4700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:31:16.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_3 2024-03-26T06:31:16.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:16 smithi170 ceph-mon[36912]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:31:16.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:16 smithi170 ceph-mon[36912]: osdmap e25: 4 total, 3 up, 4 in 2024-03-26T06:31:16.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:16 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:16.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:16 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:16.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:16 smithi170 ceph-mon[36912]: from='osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291]' entity='osd.3' 2024-03-26T06:31:16.811 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:31:17.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:31:17.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:31:17.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:31:17.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291] boot 2024-03-26T06:31:17.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T06:31:17.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:17.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi038 2024-03-26T06:31:17.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:17.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi038 to 8546M 2024-03-26T06:31:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:17.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291] boot 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi038 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi038 to 8546M 2024-03-26T06:31:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:17.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:17.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:17.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:17.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:31:17.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:31:17.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-26T06:31:17.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: osd.3 [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291] boot 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi038 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi038 to 8546M 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:18.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.219+0000 7f9d04533700 1 -- 172.21.15.121:0/759938847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc100d80 msgr2=0x7f9cfc1011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:31:18.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.219+0000 7f9d04533700 1 --2- 172.21.15.121:0/759938847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc100d80 0x7f9cfc1011a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9cf80099c0 tx=0x7f9cf8009cd0 comp rx=0 tx=0).stop 2024-03-26T06:31:18.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 -- 172.21.15.121:0/759938847 shutdown_connections 2024-03-26T06:31:18.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 --2- 172.21.15.121:0/759938847 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cfc102920 0x7f9cfc1091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:18.224 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 --2- 172.21.15.121:0/759938847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc100d80 0x7f9cfc1011a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 --2- 172.21.15.121:0/759938847 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9cfc101f80 0x7f9cfc1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 -- 172.21.15.121:0/759938847 >> 172.21.15.121:0/759938847 conn(0x7f9cfc0fc340 msgr2=0x7f9cfc0fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 -- 172.21.15.121:0/759938847 shutdown_connections 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 -- 172.21.15.121:0/759938847 wait complete. 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 Processor -- start 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 -- start start 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9cfc100d80 0x7f9cfc104d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc101f80 0x7f9cfc103360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cfc102920 0x7f9cfc1038a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc10c970 con 0x7f9cfc100d80 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc10c7f0 con 0x7f9cfc101f80 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d04533700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc10caf0 con 0x7f9cfc102920 2024-03-26T06:31:18.225 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.220+0000 7f9d01ace700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc101f80 0x7f9cfc103360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d01ace700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc101f80 0x7f9cfc103360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:56960/0 (socket says 172.21.15.121:56960) 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d01ace700 1 -- 172.21.15.121:0/413186711 learned_addr learned my addr 172.21.15.121:0/413186711 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d022cf700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9cfc100d80 0x7f9cfc104d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d02ad0700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cfc102920 0x7f9cfc1038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d01ace700 1 -- 172.21.15.121:0/413186711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cfc102920 msgr2=0x7f9cfc1038a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d01ace700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cfc102920 0x7f9cfc1038a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d01ace700 1 -- 172.21.15.121:0/413186711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9cfc100d80 msgr2=0x7f9cfc104d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d01ace700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9cfc100d80 0x7f9cfc104d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d01ace700 1 -- 172.21.15.121:0/413186711 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc104150 con 0x7f9cfc101f80 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d01ace700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc101f80 0x7f9cfc103360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9cec00b530 tx=0x7f9cec00b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9cf37fe700 1 -- 172.21.15.121:0/413186711 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec015020 con 0x7f9cfc101f80 2024-03-26T06:31:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d04533700 1 -- 172.21.15.121:0/413186711 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc1a2240 con 0x7f9cfc101f80 2024-03-26T06:31:18.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9cf37fe700 1 -- 172.21.15.121:0/413186711 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cec011d10 con 0x7f9cfc101f80 2024-03-26T06:31:18.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9d04533700 1 -- 172.21.15.121:0/413186711 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc1a2800 con 0x7f9cfc101f80 2024-03-26T06:31:18.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.221+0000 7f9cf37fe700 1 -- 172.21.15.121:0/413186711 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec00f2e0 con 0x7f9cfc101f80 2024-03-26T06:31:18.227 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.222+0000 7f9d04533700 1 -- 172.21.15.121:0/413186711 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cfc068aa0 con 0x7f9cfc101f80 2024-03-26T06:31:18.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.223+0000 7f9cf37fe700 1 -- 172.21.15.121:0/413186711 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9cec018020 con 0x7f9cfc101f80 2024-03-26T06:31:18.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.224+0000 7f9cf37fe700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ce80a2890 0x7f9ce80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:31:18.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.224+0000 7f9cf37fe700 1 -- 172.21.15.121:0/413186711 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f9cec0cd290 con 0x7f9cfc101f80 2024-03-26T06:31:18.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.224+0000 7f9d022cf700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ce80a2890 0x7f9ce80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:31:18.228 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.225+0000 7f9d022cf700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ce80a2890 0x7f9ce80a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800b580 tx=0x7f9cf8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:31:18.230 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.228+0000 7f9cf37fe700 1 -- 172.21.15.121:0/413186711 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9cec01f020 con 0x7f9cfc101f80 2024-03-26T06:31:18.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:18 smithi121 ceph-mon[36349]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T06:31:18.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:18 smithi170 ceph-mon[36912]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T06:31:18.435 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:18.433+0000 7f9d04533700 1 -- 172.21.15.121:0/413186711 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cfc0fe4c0 con 0x7f9ce80a2890 2024-03-26T06:31:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:18 smithi038 ceph-mon[32265]: osdmap e27: 4 total, 4 up, 4 in 2024-03-26T06:31:19.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:19 smithi121 ceph-mon[36349]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:19.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:19 smithi121 ceph-mon[36349]: osdmap e28: 4 total, 4 up, 4 in 2024-03-26T06:31:19.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:19 smithi121 ceph-mon[36349]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:31:19.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:31:19.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:31:19.353 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:19.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:19 smithi038 ceph-mon[32265]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:19.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:19 smithi038 ceph-mon[32265]: osdmap e28: 4 total, 4 up, 4 in 2024-03-26T06:31:19.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:19 smithi038 ceph-mon[32265]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:31:19.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:31:19.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:31:19.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:19.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:19 smithi170 ceph-mon[36912]: pgmap v103: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:19.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:19 smithi170 ceph-mon[36912]: osdmap e28: 4 total, 4 up, 4 in 2024-03-26T06:31:19.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:19 smithi170 ceph-mon[36912]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:31:19.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:31:19.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:31:19.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:20.380 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:20 smithi121 ceph-mon[36349]: osdmap e29: 4 total, 4 up, 4 in 2024-03-26T06:31:20.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:20 smithi038 ceph-mon[32265]: osdmap e29: 4 total, 4 up, 4 in 2024-03-26T06:31:20.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:20 smithi170 ceph-mon[36912]: osdmap e29: 4 total, 4 up, 4 in 2024-03-26T06:31:21.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:21 smithi038 ceph-mon[32265]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:21.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:21 smithi038 ceph-mon[32265]: osdmap e30: 4 total, 4 up, 4 in 2024-03-26T06:31:21.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:21 smithi038 ceph-mon[32265]: from='client.? 172.21.15.121:0/1415745253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14ae6f68-9b15-4098-8bea-4c1688b4a0fb"}]: dispatch 2024-03-26T06:31:21.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:21 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14ae6f68-9b15-4098-8bea-4c1688b4a0fb"}]: dispatch 2024-03-26T06:31:21.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:21 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14ae6f68-9b15-4098-8bea-4c1688b4a0fb"}]': finished 2024-03-26T06:31:21.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:21 smithi038 ceph-mon[32265]: osdmap e31: 5 total, 4 up, 5 in 2024-03-26T06:31:21.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:21 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:21.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:21 smithi121 ceph-mon[36349]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:21.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:21 smithi121 ceph-mon[36349]: osdmap e30: 4 total, 4 up, 4 in 2024-03-26T06:31:21.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:21 smithi121 ceph-mon[36349]: from='client.? 172.21.15.121:0/1415745253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14ae6f68-9b15-4098-8bea-4c1688b4a0fb"}]: dispatch 2024-03-26T06:31:21.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:21 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14ae6f68-9b15-4098-8bea-4c1688b4a0fb"}]: dispatch 2024-03-26T06:31:21.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:21 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14ae6f68-9b15-4098-8bea-4c1688b4a0fb"}]': finished 2024-03-26T06:31:21.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:21 smithi121 ceph-mon[36349]: osdmap e31: 5 total, 4 up, 5 in 2024-03-26T06:31:21.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:21 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:21.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:21 smithi170 ceph-mon[36912]: pgmap v106: 1 pgs: 1 remapped+peering; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:21.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:21 smithi170 ceph-mon[36912]: osdmap e30: 4 total, 4 up, 4 in 2024-03-26T06:31:21.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:21 smithi170 ceph-mon[36912]: from='client.? 172.21.15.121:0/1415745253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14ae6f68-9b15-4098-8bea-4c1688b4a0fb"}]: dispatch 2024-03-26T06:31:21.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:21 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14ae6f68-9b15-4098-8bea-4c1688b4a0fb"}]: dispatch 2024-03-26T06:31:21.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:21 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14ae6f68-9b15-4098-8bea-4c1688b4a0fb"}]': finished 2024-03-26T06:31:21.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:21 smithi170 ceph-mon[36912]: osdmap e31: 5 total, 4 up, 5 in 2024-03-26T06:31:21.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:21 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:22.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:22 smithi038 ceph-mon[32265]: from='client.? 172.21.15.121:0/2876887306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:31:22.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:22 smithi121 ceph-mon[36349]: from='client.? 172.21.15.121:0/2876887306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:31:22.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:22 smithi170 ceph-mon[36912]: from='client.? 172.21.15.121:0/2876887306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:31:23.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:23 smithi038 ceph-mon[32265]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:31:23.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:23 smithi121 ceph-mon[36349]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:31:23.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:23 smithi170 ceph-mon[36912]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:31:25.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:25 smithi038 ceph-mon[32265]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T06:31:25.605 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:25 smithi121 ceph-mon[36349]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T06:31:25.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:25 smithi170 ceph-mon[36912]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T06:31:27.283 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:27 smithi121 ceph-mon[36349]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-26T06:31:27.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:27 smithi038 ceph-mon[32265]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-26T06:31:27.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:27 smithi170 ceph-mon[36912]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-03-26T06:31:29.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:29 smithi038 ceph-mon[32265]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T06:31:29.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T06:31:29.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:29.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:29 smithi038 ceph-mon[32265]: Deploying daemon osd.4 on smithi121 2024-03-26T06:31:29.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:29 smithi121 ceph-mon[36349]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T06:31:29.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T06:31:29.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:29.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:29 smithi121 ceph-mon[36349]: Deploying daemon osd.4 on smithi121 2024-03-26T06:31:29.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:29 smithi170 ceph-mon[36912]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T06:31:29.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T06:31:29.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:29.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:29 smithi170 ceph-mon[36912]: Deploying daemon osd.4 on smithi121 2024-03-26T06:31:31.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:31 smithi038 ceph-mon[32265]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T06:31:31.599 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:31 smithi121 ceph-mon[36349]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T06:31:31.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:31 smithi170 ceph-mon[36912]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T06:31:33.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:33 smithi121 ceph-mon[36349]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T06:31:33.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:33 smithi038 ceph-mon[32265]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T06:31:33.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:33 smithi170 ceph-mon[36912]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T06:31:34.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:34 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:34.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:34 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:34.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:34 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:35.065 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:34 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:35.065 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:34 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:35.065 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:34 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:35.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:34 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:35.082 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:34 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:35.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:34 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:35.803 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:35 smithi121 ceph-mon[36349]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:36.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:35 smithi038 ceph-mon[32265]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:36.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:35 smithi170 ceph-mon[36912]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:37.255 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.252+0000 7f9cf37fe700 1 -- 172.21.15.121:0/413186711 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9cfc0fe4c0 con 0x7f9ce80a2890 2024-03-26T06:31:37.255 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 4 on host 'smithi121' 2024-03-26T06:31:37.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 -- 172.21.15.121:0/413186711 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ce80a2890 msgr2=0x7f9ce80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:31:37.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ce80a2890 0x7f9ce80a4d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9cf800b580 tx=0x7f9cf8005fb0 comp rx=0 tx=0).stop 2024-03-26T06:31:37.257 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 -- 172.21.15.121:0/413186711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc101f80 msgr2=0x7f9cfc103360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:31:37.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc101f80 0x7f9cfc103360 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9cec00b530 tx=0x7f9cec00b840 comp rx=0 tx=0).stop 2024-03-26T06:31:37.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 -- 172.21.15.121:0/413186711 shutdown_connections 2024-03-26T06:31:37.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9cfc102920 0x7f9cfc1038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:37.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9cfc100d80 0x7f9cfc104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:37.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ce80a2890 0x7f9ce80a4d50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:37.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 --2- 172.21.15.121:0/413186711 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cfc101f80 0x7f9cfc103360 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:37.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 -- 172.21.15.121:0/413186711 >> 172.21.15.121:0/413186711 conn(0x7f9cfc0fc340 msgr2=0x7f9cfc0fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:31:37.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 -- 172.21.15.121:0/413186711 shutdown_connections 2024-03-26T06:31:37.258 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:31:37.255+0000 7f9d04533700 1 -- 172.21.15.121:0/413186711 wait complete. 2024-03-26T06:31:37.552 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:37 smithi121 ceph-mon[36349]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:37.552 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.552 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:37.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:37.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.553 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:37.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:37 smithi170 ceph-mon[36912]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:37.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:37.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:37.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:37.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:37 smithi038 ceph-mon[32265]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:37.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:37.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:37.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:37.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:37.847 DEBUG:teuthology.orchestra.run.smithi121:osd.4> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T06:31:37.849 INFO:tasks.cephadm:Deploying osd.5 on smithi170 with /dev/vg_nvme/lv_3... 2024-03-26T06:31:37.849 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T06:31:38.094 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:31:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:38 smithi121 ceph-mon[36349]: from='osd.4 [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T06:31:38.422 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:38 smithi121 ceph-mon[36349]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T06:31:38.498 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:38 smithi170 ceph-mon[36912]: from='osd.4 [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T06:31:38.498 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:38 smithi170 ceph-mon[36912]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T06:31:38.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:38 smithi038 ceph-mon[32265]: from='osd.4 [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T06:31:38.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:38 smithi038 ceph-mon[32265]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T06:31:39.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:39 smithi121 ceph-mon[36349]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:39.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:39 smithi121 ceph-mon[36349]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T06:31:39.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:39 smithi121 ceph-mon[36349]: osdmap e32: 5 total, 4 up, 5 in 2024-03-26T06:31:39.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:39.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:39 smithi121 ceph-mon[36349]: from='osd.4 [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:31:39.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:39 smithi121 ceph-mon[36349]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:31:39.646 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:31:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4[47510]: 2024-03-26T06:31:39.393+0000 7f310df97700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T06:31:39.646 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:31:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4[47510]: 2024-03-26T06:31:39.400+0000 7f310a58b700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:31:39.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:39 smithi170 ceph-mon[36912]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:39.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:39 smithi170 ceph-mon[36912]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T06:31:39.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:39 smithi170 ceph-mon[36912]: osdmap e32: 5 total, 4 up, 5 in 2024-03-26T06:31:39.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:39.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:39 smithi170 ceph-mon[36912]: from='osd.4 [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:31:39.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:39 smithi170 ceph-mon[36912]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:31:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:39 smithi038 ceph-mon[32265]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:39 smithi038 ceph-mon[32265]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T06:31:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:39 smithi038 ceph-mon[32265]: osdmap e32: 5 total, 4 up, 5 in 2024-03-26T06:31:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:39 smithi038 ceph-mon[32265]: from='osd.4 [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:31:39.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:39 smithi038 ceph-mon[32265]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:31:39.933 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T06:31:39.934 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T06:31:39.934 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-26T06:31:39.934 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-26T06:31:39.934 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00854648 s, 1.2 GB/s 2024-03-26T06:31:39.934 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-26T06:31:40.462 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_3 2024-03-26T06:31:40.488 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:40 smithi170 ceph-mon[36912]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:31:40.488 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:40 smithi170 ceph-mon[36912]: osdmap e33: 5 total, 4 up, 5 in 2024-03-26T06:31:40.488 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:40 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:40.488 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:40 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:40.707 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:31:40.710 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:40 smithi121 ceph-mon[36349]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:31:40.710 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:40 smithi121 ceph-mon[36349]: osdmap e33: 5 total, 4 up, 5 in 2024-03-26T06:31:40.710 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:40 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:40.711 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:40 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:40.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:40 smithi038 ceph-mon[32265]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:31:40.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:40 smithi038 ceph-mon[32265]: osdmap e33: 5 total, 4 up, 5 in 2024-03-26T06:31:40.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:40 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:40.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:40 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:41.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:31:41.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:31:41.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:41.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:41.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: osd.4 [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176] boot 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi121 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi121 to 8540M 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:41 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:41.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:31:41.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:31:41.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:41.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:41.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: osd.4 [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176] boot 2024-03-26T06:31:41.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T06:31:41.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:41.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi121 2024-03-26T06:31:41.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:41.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:41.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:41.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:41.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi121 to 8540M 2024-03-26T06:31:41.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:41.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:41.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:41.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:41 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:41.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:31:41.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:31:41.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-26T06:31:41.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:41.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: osd.4 [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176] boot 2024-03-26T06:31:41.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T06:31:41.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:31:41.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi121 2024-03-26T06:31:41.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:41.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:41.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:41.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:31:41.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi121 to 8540M 2024-03-26T06:31:41.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:41.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:41.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:31:41.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:41 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:42.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.253+0000 7f37f63cb700 1 -- 172.21.15.170:0/1975127106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f0105240 msgr2=0x7f37f0107630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:31:42.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.253+0000 7f37f63cb700 1 --2- 172.21.15.170:0/1975127106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f0105240 0x7f37f0107630 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009990 tx=0x7f37e4009ca0 comp rx=0 tx=0).stop 2024-03-26T06:31:42.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 -- 172.21.15.170:0/1975127106 shutdown_connections 2024-03-26T06:31:42.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 --2- 172.21.15.170:0/1975127106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37f0107b70 0x7f37f0109fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:42.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 --2- 172.21.15.170:0/1975127106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0102910 0x7f37f0104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:42.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 --2- 172.21.15.170:0/1975127106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f0105240 0x7f37f0107630 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:42.258 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 -- 172.21.15.170:0/1975127106 >> 172.21.15.170:0/1975127106 conn(0x7f37f00fc340 msgr2=0x7f37f00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 -- 172.21.15.170:0/1975127106 shutdown_connections 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 -- 172.21.15.170:0/1975127106 wait complete. 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 Processor -- start 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 -- start start 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f0102910 0x7f37f01980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37f0105240 0x7f37f01985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0107b70 0x7f37f019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f010da60 con 0x7f37f0105240 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f010d8e0 con 0x7f37f0107b70 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.254+0000 7f37f63cb700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f010dbe0 con 0x7f37f0102910 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37effff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f0102910 0x7f37f01980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:31:42.259 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37effff700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f0102910 0x7f37f01980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:46236/0 (socket says 172.21.15.170:46236) 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37effff700 1 -- 172.21.15.170:0/1705237106 learned_addr learned my addr 172.21.15.170:0/1705237106 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37ef7fe700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37f0105240 0x7f37f01985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37f4968700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0107b70 0x7f37f019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37effff700 1 -- 172.21.15.170:0/1705237106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0107b70 msgr2=0x7f37f019f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37effff700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0107b70 0x7f37f019f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37effff700 1 -- 172.21.15.170:0/1705237106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37f0105240 msgr2=0x7f37f01985e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37effff700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37f0105240 0x7f37f01985e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37effff700 1 -- 172.21.15.170:0/1705237106 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f019fd00 con 0x7f37f0102910 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37effff700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f0102910 0x7f37f01980a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37e000e9f0 tx=0x7f37e000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37ed7fa700 1 -- 172.21.15.170:0/1705237106 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e000cbd0 con 0x7f37f0102910 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37ed7fa700 1 -- 172.21.15.170:0/1705237106 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37e000ee90 con 0x7f37f0102910 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37ed7fa700 1 -- 172.21.15.170:0/1705237106 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e00056c0 con 0x7f37f0102910 2024-03-26T06:31:42.260 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37f63cb700 1 -- 172.21.15.170:0/1705237106 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f019fff0 con 0x7f37f0102910 2024-03-26T06:31:42.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.255+0000 7f37f63cb700 1 -- 172.21.15.170:0/1705237106 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f01a03e0 con 0x7f37f0102910 2024-03-26T06:31:42.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.256+0000 7f37f63cb700 1 -- 172.21.15.170:0/1705237106 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f0068aa0 con 0x7f37f0102910 2024-03-26T06:31:42.261 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.257+0000 7f37ed7fa700 1 -- 172.21.15.170:0/1705237106 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f37e0004750 con 0x7f37f0102910 2024-03-26T06:31:42.262 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.258+0000 7f37ed7fa700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f37d80a2890 0x7f37d80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:31:42.262 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.258+0000 7f37ed7fa700 1 -- 172.21.15.170:0/1705237106 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f37e00cdc50 con 0x7f37f0102910 2024-03-26T06:31:42.262 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.258+0000 7f37ef7fe700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f37d80a2890 0x7f37d80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:31:42.262 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.258+0000 7f37ef7fe700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f37d80a2890 0x7f37d80a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f37e400b2e0 tx=0x7f37e400b910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:31:42.265 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.262+0000 7f37ed7fa700 1 -- 172.21.15.170:0/1705237106 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f37e00d3020 con 0x7f37f0102910 2024-03-26T06:31:42.466 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:31:42.464+0000 7f37f63cb700 1 -- 172.21.15.170:0/1705237106 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f37f0062220 con 0x7f37d80a2890 2024-03-26T06:31:42.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:42 smithi170 ceph-mon[36912]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T06:31:42.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:42 smithi038 ceph-mon[32265]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T06:31:42.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:42 smithi121 ceph-mon[36349]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T06:31:43.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:43 smithi170 ceph-mon[36912]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:43.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:43 smithi170 ceph-mon[36912]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:31:43.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:31:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:31:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:43 smithi038 ceph-mon[32265]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:43 smithi038 ceph-mon[32265]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:31:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:31:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:31:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:43.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:43 smithi121 ceph-mon[36349]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:43.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:43 smithi121 ceph-mon[36349]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:31:43.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:31:43.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:31:43.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:45 smithi038 ceph-mon[32265]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:45 smithi038 ceph-mon[32265]: from='client.? 172.21.15.170:0/715269377' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb8ecf7-e48a-4e5a-b313-f84660cf29f6"}]: dispatch 2024-03-26T06:31:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:45 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb8ecf7-e48a-4e5a-b313-f84660cf29f6"}]: dispatch 2024-03-26T06:31:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:45 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bb8ecf7-e48a-4e5a-b313-f84660cf29f6"}]': finished 2024-03-26T06:31:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:45 smithi038 ceph-mon[32265]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T06:31:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:45 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:31:45.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:45 smithi121 ceph-mon[36349]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:45.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:45 smithi121 ceph-mon[36349]: from='client.? 172.21.15.170:0/715269377' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb8ecf7-e48a-4e5a-b313-f84660cf29f6"}]: dispatch 2024-03-26T06:31:45.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:45 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb8ecf7-e48a-4e5a-b313-f84660cf29f6"}]: dispatch 2024-03-26T06:31:45.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:45 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bb8ecf7-e48a-4e5a-b313-f84660cf29f6"}]': finished 2024-03-26T06:31:45.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:45 smithi121 ceph-mon[36349]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T06:31:45.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:45 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:31:45.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:45 smithi170 ceph-mon[36912]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:45.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:45 smithi170 ceph-mon[36912]: from='client.? 172.21.15.170:0/715269377' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb8ecf7-e48a-4e5a-b313-f84660cf29f6"}]: dispatch 2024-03-26T06:31:45.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:45 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8bb8ecf7-e48a-4e5a-b313-f84660cf29f6"}]: dispatch 2024-03-26T06:31:45.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:45 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8bb8ecf7-e48a-4e5a-b313-f84660cf29f6"}]': finished 2024-03-26T06:31:45.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:45 smithi170 ceph-mon[36912]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T06:31:45.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:45 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:31:46.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:46 smithi038 ceph-mon[32265]: from='client.? 172.21.15.170:0/2854400797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:31:46.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:46 smithi121 ceph-mon[36349]: from='client.? 172.21.15.170:0/2854400797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:31:46.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:46 smithi170 ceph-mon[36912]: from='client.? 172.21.15.170:0/2854400797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:31:47.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:47 smithi038 ceph-mon[32265]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:47.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:47 smithi121 ceph-mon[36349]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:47.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:47 smithi170 ceph-mon[36912]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:49.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:49 smithi038 ceph-mon[32265]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:49.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:49 smithi121 ceph-mon[36349]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:49.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:49 smithi170 ceph-mon[36912]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:51.624 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:51 smithi170 ceph-mon[36912]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:51.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:51 smithi038 ceph-mon[32265]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:51.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:51 smithi121 ceph-mon[36349]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:52.822 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T06:31:52.823 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:52.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T06:31:52.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:52.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T06:31:52.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:31:53.753 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:53 smithi170 ceph-mon[36912]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:53.753 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:53 smithi170 ceph-mon[36912]: Deploying daemon osd.5 on smithi170 2024-03-26T06:31:53.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:53 smithi038 ceph-mon[32265]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:53.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:53 smithi038 ceph-mon[32265]: Deploying daemon osd.5 on smithi170 2024-03-26T06:31:53.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:53 smithi121 ceph-mon[36349]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:53.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:53 smithi121 ceph-mon[36349]: Deploying daemon osd.5 on smithi170 2024-03-26T06:31:55.825 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:55 smithi170 ceph-mon[36912]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:55.832 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:55 smithi038 ceph-mon[32265]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:55.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:55 smithi121 ceph-mon[36349]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:57.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:57 smithi170 ceph-mon[36912]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:57.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:57.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:57.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:57 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:57 smithi038 ceph-mon[32265]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:57 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:57.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:57 smithi121 ceph-mon[36349]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:57.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:31:57.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:57.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:57 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:31:59.757 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:31:59 smithi170 ceph-mon[36912]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:59.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:31:59 smithi038 ceph-mon[32265]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:31:59.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:31:59 smithi121 ceph-mon[36349]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:32:01.287 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 5 on host 'smithi170' 2024-03-26T06:32:01.287 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.284+0000 7f37ed7fa700 1 -- 172.21.15.170:0/1705237106 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f37f0062220 con 0x7f37d80a2890 2024-03-26T06:32:01.288 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.286+0000 7f37f63cb700 1 -- 172.21.15.170:0/1705237106 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f37d80a2890 msgr2=0x7f37d80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:01.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.286+0000 7f37f63cb700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f37d80a2890 0x7f37d80a4d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f37e400b2e0 tx=0x7f37e400b910 comp rx=0 tx=0).stop 2024-03-26T06:32:01.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.286+0000 7f37f63cb700 1 -- 172.21.15.170:0/1705237106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f0102910 msgr2=0x7f37f01980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:01.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.286+0000 7f37f63cb700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f0102910 0x7f37f01980a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f37e000e9f0 tx=0x7f37e000ed00 comp rx=0 tx=0).stop 2024-03-26T06:32:01.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.287+0000 7f37f63cb700 1 -- 172.21.15.170:0/1705237106 shutdown_connections 2024-03-26T06:32:01.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.287+0000 7f37f63cb700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f37d80a2890 0x7f37d80a4d50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:01.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.287+0000 7f37f63cb700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f0107b70 0x7f37f019f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:01.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.287+0000 7f37f63cb700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f37f0102910 0x7f37f01980a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:01.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.287+0000 7f37f63cb700 1 --2- 172.21.15.170:0/1705237106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37f0105240 0x7f37f01985e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:01.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.287+0000 7f37f63cb700 1 -- 172.21.15.170:0/1705237106 >> 172.21.15.170:0/1705237106 conn(0x7f37f00fc340 msgr2=0x7f37f0103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:32:01.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.287+0000 7f37f63cb700 1 -- 172.21.15.170:0/1705237106 shutdown_connections 2024-03-26T06:32:01.289 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:01.287+0000 7f37f63cb700 1 -- 172.21.15.170:0/1705237106 wait complete. 2024-03-26T06:32:01.633 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:01 smithi170 ceph-mon[36912]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:32:01.633 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:01 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.633 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:01 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.633 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:01 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:01.633 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:01 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:01.633 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:01 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.633 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:01 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.633 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:01 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.633 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:01 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:01.633 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:01 smithi170 ceph-mon[36912]: from='osd.5 [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T06:32:01.633 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:01 smithi170 ceph-mon[36912]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T06:32:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:01 smithi038 ceph-mon[32265]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:32:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:01.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:01 smithi038 ceph-mon[32265]: from='osd.5 [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T06:32:01.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:01 smithi038 ceph-mon[32265]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T06:32:01.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:01 smithi121 ceph-mon[36349]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:32:01.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:01 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:01 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:01 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:01.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:01 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:01.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:01 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:01 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:01 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:01.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:01 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:01.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:01 smithi121 ceph-mon[36349]: from='osd.5 [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T06:32:01.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:01 smithi121 ceph-mon[36349]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T06:32:01.954 DEBUG:teuthology.orchestra.run.smithi170:osd.5> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T06:32:01.996 INFO:tasks.cephadm:Deploying osd.6 on smithi038 with /dev/vg_nvme/lv_2... 2024-03-26T06:32:01.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T06:32:02.249 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:32:03.406 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:03 smithi170 ceph-mon[36912]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:32:03.407 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:03 smithi170 ceph-mon[36912]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T06:32:03.407 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:03 smithi170 ceph-mon[36912]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T06:32:03.407 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:03.407 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:03 smithi170 ceph-mon[36912]: from='osd.5 [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:32:03.407 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:03 smithi170 ceph-mon[36912]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:32:03.407 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:32:03 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:32:03.261+0000 7f60686d6700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T06:32:03.407 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:32:03 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:32:03.270+0000 7f6064cca700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:32:03.547 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:03 smithi038 ceph-mon[32265]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:32:03.547 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:03 smithi038 ceph-mon[32265]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T06:32:03.547 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:03 smithi038 ceph-mon[32265]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T06:32:03.547 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:03.547 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:03 smithi038 ceph-mon[32265]: from='osd.5 [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:32:03.547 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:03 smithi038 ceph-mon[32265]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:32:03.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:03 smithi121 ceph-mon[36349]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:32:03.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:03 smithi121 ceph-mon[36349]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T06:32:03.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:03 smithi121 ceph-mon[36349]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T06:32:03.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:03.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:03 smithi121 ceph-mon[36349]: from='osd.5 [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:32:03.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:03 smithi121 ceph-mon[36349]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:32:04.271 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T06:32:04.272 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T06:32:04.272 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-03-26T06:32:04.272 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-03-26T06:32:04.272 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0136315 s, 769 MB/s 2024-03-26T06:32:04.272 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-03-26T06:32:04.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:04 smithi170 ceph-mon[36912]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:32:04.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:04 smithi170 ceph-mon[36912]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T06:32:04.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:04 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:04.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:04 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:04.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:04 smithi170 ceph-mon[36912]: osd.5 [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153] boot 2024-03-26T06:32:04.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:04 smithi170 ceph-mon[36912]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T06:32:04.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:04 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:04 smithi038 ceph-mon[32265]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:32:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:04 smithi038 ceph-mon[32265]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T06:32:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:04.580 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:04.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:04 smithi038 ceph-mon[32265]: osd.5 [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153] boot 2024-03-26T06:32:04.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:04 smithi038 ceph-mon[32265]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T06:32:04.581 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:04 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:04.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:04 smithi121 ceph-mon[36349]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:32:04.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:04 smithi121 ceph-mon[36349]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T06:32:04.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:04 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:04.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:04 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:04.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:04 smithi121 ceph-mon[36349]: osd.5 [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153] boot 2024-03-26T06:32:04.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:04 smithi121 ceph-mon[36349]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T06:32:04.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:04 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:32:04.917 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi038:vg_nvme/lv_2 2024-03-26T06:32:05.163 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:32:05.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:32:05.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:32:05.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:32:05.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi170 2024-03-26T06:32:05.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:05.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:05.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:05.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:05.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi170 to 8540M 2024-03-26T06:32:05.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:05.568 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:05.569 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:05.569 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:32:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:32:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:32:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi170 2024-03-26T06:32:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi170 to 8540M 2024-03-26T06:32:05.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:05.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:05.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:05.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:05.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:32:05.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:32:05.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 151 MiB used, 447 GiB / 447 GiB avail 2024-03-26T06:32:05.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi170 2024-03-26T06:32:05.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:05.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:05.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:05.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:05.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi170 to 8540M 2024-03-26T06:32:05.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:05.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:05.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:05.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:06.507 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:06 smithi038 ceph-mon[32265]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T06:32:06.507 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:06 smithi038 ceph-mon[32265]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T06:32:06.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:06 smithi121 ceph-mon[36349]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T06:32:06.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:06 smithi121 ceph-mon[36349]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T06:32:06.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:06 smithi170 ceph-mon[36912]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T06:32:06.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:06 smithi170 ceph-mon[36912]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T06:32:06.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.731+0000 7f6181abe700 1 -- 172.21.15.38:0/3692651996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f617c102f70 msgr2=0x7f617c1033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:06.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.731+0000 7f6181abe700 1 --2- 172.21.15.38:0/3692651996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f617c102f70 0x7f617c1033f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6170009990 tx=0x7f6170009ca0 comp rx=0 tx=0).stop 2024-03-26T06:32:06.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.731+0000 7f6181abe700 1 -- 172.21.15.38:0/3692651996 shutdown_connections 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.731+0000 7f6181abe700 1 --2- 172.21.15.38:0/3692651996 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c103930 0x7f617c10a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.731+0000 7f6181abe700 1 --2- 172.21.15.38:0/3692651996 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f617c101e10 0x7f617c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.731+0000 7f6181abe700 1 --2- 172.21.15.38:0/3692651996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f617c102f70 0x7f617c1033f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.731+0000 7f6181abe700 1 -- 172.21.15.38:0/3692651996 >> 172.21.15.38:0/3692651996 conn(0x7f617c0fd3f0 msgr2=0x7f617c0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.732+0000 7f6181abe700 1 -- 172.21.15.38:0/3692651996 shutdown_connections 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.732+0000 7f6181abe700 1 -- 172.21.15.38:0/3692651996 wait complete. 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.732+0000 7f6181abe700 1 Processor -- start 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.732+0000 7f6181abe700 1 -- start start 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.732+0000 7f6181abe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f617c101e10 0x7f617c197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.732+0000 7f6181abe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c102f70 0x7f617c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.732+0000 7f6181abe700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f617c103930 0x7f617c19f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f6181abe700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c10da00 con 0x7f617c101e10 2024-03-26T06:32:06.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f6181abe700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c10d880 con 0x7f617c102f70 2024-03-26T06:32:06.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f6181abe700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f617c10db80 con 0x7f617c103930 2024-03-26T06:32:06.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f617c101e10 0x7f617c197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:06.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f617c101e10 0x7f617c197ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:34974/0 (socket says 172.21.15.38:34974) 2024-03-26T06:32:06.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617b7fe700 1 -- 172.21.15.38:0/2366585806 learned_addr learned my addr 172.21.15.38:0/2366585806 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:32:06.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617affd700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c102f70 0x7f617c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:06.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617bfff700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f617c103930 0x7f617c19f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:06.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617b7fe700 1 -- 172.21.15.38:0/2366585806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f617c103930 msgr2=0x7f617c19f5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:06.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617b7fe700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f617c103930 0x7f617c19f5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:06.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617b7fe700 1 -- 172.21.15.38:0/2366585806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c102f70 msgr2=0x7f617c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:06.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617b7fe700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c102f70 0x7f617c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:06.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617b7fe700 1 -- 172.21.15.38:0/2366585806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c19fc50 con 0x7f617c101e10 2024-03-26T06:32:06.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617bfff700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f617c103930 0x7f617c19f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:32:06.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f617b7fe700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f617c101e10 0x7f617c197ff0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f616c00c9a0 tx=0x7f616c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:32:06.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f6178ff9700 1 -- 172.21.15.38:0/2366585806 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c007a60 con 0x7f617c101e10 2024-03-26T06:32:06.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f6178ff9700 1 -- 172.21.15.38:0/2366585806 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f616c004d10 con 0x7f617c101e10 2024-03-26T06:32:06.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f6181abe700 1 -- 172.21.15.38:0/2366585806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c19ff40 con 0x7f617c101e10 2024-03-26T06:32:06.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.733+0000 7f6178ff9700 1 -- 172.21.15.38:0/2366585806 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f616c005690 con 0x7f617c101e10 2024-03-26T06:32:06.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.734+0000 7f6181abe700 1 -- 172.21.15.38:0/2366585806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c1a0390 con 0x7f617c101e10 2024-03-26T06:32:06.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.734+0000 7f6181abe700 1 -- 172.21.15.38:0/2366585806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c068aa0 con 0x7f617c101e10 2024-03-26T06:32:06.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.735+0000 7f6178ff9700 1 -- 172.21.15.38:0/2366585806 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f616c007c00 con 0x7f617c101e10 2024-03-26T06:32:06.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.736+0000 7f6178ff9700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f61640a2910 0x7f61640a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:06.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.736+0000 7f617affd700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f61640a2910 0x7f61640a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:06.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.736+0000 7f6178ff9700 1 -- 172.21.15.38:0/2366585806 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f616c0cd910 con 0x7f617c101e10 2024-03-26T06:32:06.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.737+0000 7f617affd700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f61640a2910 0x7f61640a4dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f617000b5c0 tx=0x7f61700038f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:32:06.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.741+0000 7f6178ff9700 1 -- 172.21.15.38:0/2366585806 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f616c09b8e0 con 0x7f617c101e10 2024-03-26T06:32:06.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:06.954+0000 7f6181abe700 1 -- 172.21.15.38:0/2366585806 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f617c062220 con 0x7f61640a2910 2024-03-26T06:32:07.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:07 smithi121 ceph-mon[36349]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:07.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:07 smithi121 ceph-mon[36349]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:32:07.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:07 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:32:07.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:07 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:32:07.546 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:07 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:07.568 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:07 smithi170 ceph-mon[36912]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:07.568 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:07 smithi170 ceph-mon[36912]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:32:07.568 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:07 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:32:07.568 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:07 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:32:07.568 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:07 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:07.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:07 smithi038 ceph-mon[32265]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:07.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:07 smithi038 ceph-mon[32265]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:32:07.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:07 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:32:07.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:07 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:32:07.574 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:07 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:08.563 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:08 smithi121 ceph-mon[36349]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T06:32:08.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:08 smithi038 ceph-mon[32265]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T06:32:08.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:08 smithi170 ceph-mon[36912]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T06:32:09.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:09 smithi038 ceph-mon[32265]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:32:09.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:09 smithi038 ceph-mon[32265]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T06:32:09.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:09 smithi121 ceph-mon[36349]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:32:09.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:09 smithi121 ceph-mon[36349]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T06:32:09.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:09 smithi170 ceph-mon[36912]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:32:09.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:09 smithi170 ceph-mon[36912]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T06:32:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:10 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3979868306' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25f7a3a4-3f5f-4625-8f87-766305b481c4"}]: dispatch 2024-03-26T06:32:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:10 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3979868306' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25f7a3a4-3f5f-4625-8f87-766305b481c4"}]': finished 2024-03-26T06:32:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:10 smithi038 ceph-mon[32265]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T06:32:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:10 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:10 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/76656613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:32:10.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:10 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3979868306' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25f7a3a4-3f5f-4625-8f87-766305b481c4"}]: dispatch 2024-03-26T06:32:10.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:10 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3979868306' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25f7a3a4-3f5f-4625-8f87-766305b481c4"}]': finished 2024-03-26T06:32:10.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:10 smithi121 ceph-mon[36349]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T06:32:10.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:10 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:10.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:10 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/76656613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:32:10.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:10 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3979868306' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25f7a3a4-3f5f-4625-8f87-766305b481c4"}]: dispatch 2024-03-26T06:32:10.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:10 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3979868306' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25f7a3a4-3f5f-4625-8f87-766305b481c4"}]': finished 2024-03-26T06:32:10.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:10 smithi170 ceph-mon[36912]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T06:32:10.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:10 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:10.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:10 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/76656613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:32:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:11 smithi038 ceph-mon[32265]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:32:12.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:11 smithi121 ceph-mon[36349]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:32:12.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:11 smithi170 ceph-mon[36912]: pgmap v146: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:32:14.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:13 smithi038 ceph-mon[32265]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-26T06:32:14.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:13 smithi121 ceph-mon[36349]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-26T06:32:14.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:13 smithi170 ceph-mon[36912]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-26T06:32:15.897 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:15 smithi038 ceph-mon[32265]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:16.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:15 smithi121 ceph-mon[36349]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:16.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:15 smithi170 ceph-mon[36912]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:18.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:17 smithi038 ceph-mon[32265]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:18.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:17 smithi121 ceph-mon[36349]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:18.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:17 smithi170 ceph-mon[36912]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:19.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:18 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T06:32:19.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:18 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:19.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:18 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T06:32:19.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:18 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:19.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:18 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T06:32:19.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:18 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:19 smithi038 ceph-mon[32265]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:19 smithi038 ceph-mon[32265]: Deploying daemon osd.6 on smithi038 2024-03-26T06:32:20.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:19 smithi121 ceph-mon[36349]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:20.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:19 smithi121 ceph-mon[36349]: Deploying daemon osd.6 on smithi038 2024-03-26T06:32:20.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:19 smithi170 ceph-mon[36912]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:20.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:19 smithi170 ceph-mon[36912]: Deploying daemon osd.6 on smithi038 2024-03-26T06:32:21.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:21 smithi038 ceph-mon[32265]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:22.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:21 smithi121 ceph-mon[36349]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:22.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:21 smithi170 ceph-mon[36912]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:24.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:23 smithi038 ceph-mon[32265]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:24.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:24.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:24.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:24.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:23 smithi121 ceph-mon[36349]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:24.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:24.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:24.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:24.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:23 smithi170 ceph-mon[36912]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:24.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:24.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:24.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:25.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:25 smithi038 ceph-mon[32265]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:26.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:25 smithi121 ceph-mon[36349]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:26.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:25 smithi170 ceph-mon[36912]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:27.518 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 6 on host 'smithi038' 2024-03-26T06:32:27.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.515+0000 7f6178ff9700 1 -- 172.21.15.38:0/2366585806 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f617c062220 con 0x7f61640a2910 2024-03-26T06:32:27.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.517+0000 7f6181abe700 1 -- 172.21.15.38:0/2366585806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f61640a2910 msgr2=0x7f61640a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:27.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.517+0000 7f6181abe700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f61640a2910 0x7f61640a4dd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f617000b5c0 tx=0x7f61700038f0 comp rx=0 tx=0).stop 2024-03-26T06:32:27.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.518+0000 7f6181abe700 1 -- 172.21.15.38:0/2366585806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f617c101e10 msgr2=0x7f617c197ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:27.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.518+0000 7f6181abe700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f617c101e10 0x7f617c197ff0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f616c00c9a0 tx=0x7f616c00ccb0 comp rx=0 tx=0).stop 2024-03-26T06:32:27.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.518+0000 7f6181abe700 1 -- 172.21.15.38:0/2366585806 shutdown_connections 2024-03-26T06:32:27.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.518+0000 7f6181abe700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f617c103930 0x7f617c19f5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:27.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.518+0000 7f6181abe700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f617c101e10 0x7f617c197ff0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:27.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.518+0000 7f6181abe700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f61640a2910 0x7f61640a4dd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:27.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.518+0000 7f6181abe700 1 --2- 172.21.15.38:0/2366585806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f617c102f70 0x7f617c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:27.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.518+0000 7f6181abe700 1 -- 172.21.15.38:0/2366585806 >> 172.21.15.38:0/2366585806 conn(0x7f617c0fd3f0 msgr2=0x7f617c106300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:32:27.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.518+0000 7f6181abe700 1 -- 172.21.15.38:0/2366585806 shutdown_connections 2024-03-26T06:32:27.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:32:27.518+0000 7f6181abe700 1 -- 172.21.15.38:0/2366585806 wait complete. 2024-03-26T06:32:27.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:27 smithi038 ceph-mon[32265]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:27.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:27.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:27.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:27 smithi038 ceph-mon[32265]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T06:32:27.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:27.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:27 smithi121 ceph-mon[36349]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:27.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:27.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:27.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:27 smithi121 ceph-mon[36349]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T06:32:27.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:27.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:27 smithi170 ceph-mon[36912]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:27.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:27 smithi170 ceph-mon[36912]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T06:32:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:28.162 DEBUG:teuthology.orchestra.run.smithi038:osd.6> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T06:32:28.165 INFO:tasks.cephadm:Deploying osd.7 on smithi121 with /dev/vg_nvme/lv_2... 2024-03-26T06:32:28.166 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T06:32:28.410 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:32:28.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:28 smithi121 ceph-mon[36349]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T06:32:28.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:28 smithi121 ceph-mon[36349]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T06:32:28.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:28 smithi121 ceph-mon[36349]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:32:28.875 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:28 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:29.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:28 smithi038 ceph-mon[32265]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T06:32:29.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:28 smithi038 ceph-mon[32265]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T06:32:29.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:28 smithi038 ceph-mon[32265]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:32:29.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:28 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:29.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:28 smithi170 ceph-mon[36912]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T06:32:29.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:28 smithi170 ceph-mon[36912]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T06:32:29.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:28 smithi170 ceph-mon[36912]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:32:29.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:28 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:29.583 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:32:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:32:29.465+0000 7fdafec3d700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T06:32:29.583 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:32:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:32:29.477+0000 7fdaf7229700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:32:29.966 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:29 smithi121 ceph-mon[36349]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:29.966 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:29 smithi121 ceph-mon[36349]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:32:29.967 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:29 smithi121 ceph-mon[36349]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T06:32:29.967 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:29.967 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:29 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:29.967 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:29 smithi121 ceph-mon[36349]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' 2024-03-26T06:32:30.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:29 smithi038 ceph-mon[32265]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:30.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:29 smithi038 ceph-mon[32265]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:32:30.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:29 smithi038 ceph-mon[32265]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T06:32:30.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:30.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:29 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:30.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:29 smithi038 ceph-mon[32265]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' 2024-03-26T06:32:30.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:29 smithi170 ceph-mon[36912]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:30.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:29 smithi170 ceph-mon[36912]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:32:30.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:29 smithi170 ceph-mon[36912]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T06:32:30.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:30.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:29 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:30.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:29 smithi170 ceph-mon[36912]: from='osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059]' entity='osd.6' 2024-03-26T06:32:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T06:32:30.547 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T06:32:30.548 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-03-26T06:32:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-03-26T06:32:30.548 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112499 s, 932 MB/s 2024-03-26T06:32:30.548 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-03-26T06:32:31.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:30 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:32:31.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:30 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:32:31.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:30 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:31.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:30 smithi038 ceph-mon[32265]: osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059] boot 2024-03-26T06:32:31.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:30 smithi038 ceph-mon[32265]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T06:32:31.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:30 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:31.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:30 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:32:31.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:30 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:32:31.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:30 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:31.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:30 smithi121 ceph-mon[36349]: osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059] boot 2024-03-26T06:32:31.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:30 smithi121 ceph-mon[36349]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T06:32:31.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:30 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:31.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:30 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:32:31.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:30 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:32:31.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:30 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:31.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:30 smithi170 ceph-mon[36912]: osd.6 [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059] boot 2024-03-26T06:32:31.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:30 smithi170 ceph-mon[36912]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T06:32:31.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:30 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:32:31.221 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_2 2024-03-26T06:32:31.466 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:32:31.894 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:31 smithi121 ceph-mon[36349]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:31.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:31 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:31.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:31 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:31.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:31 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:31.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:31 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:31.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:31 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:31.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:31 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:31.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:31 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:31.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:31 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:31.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:31 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:32.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:31 smithi038 ceph-mon[32265]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:32.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:31 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:32.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:31 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:32.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:31 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:32.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:31 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:32.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:31 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:32.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:31 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:32.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:31 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:32.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:31 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:32.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:31 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:32.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:31 smithi170 ceph-mon[36912]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-26T06:32:32.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:31 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:32.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:31 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:32.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:31 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:32.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:31 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:32.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:31 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:32.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:31 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:32.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:31 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:32.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:31 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:32.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:31 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:32.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:32 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi038 2024-03-26T06:32:32.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:32 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi038 to 5697M 2024-03-26T06:32:32.887 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:32 smithi121 ceph-mon[36349]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T06:32:32.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.987+0000 7fb35e8d2700 1 -- 172.21.15.121:0/3320280405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3581004e0 msgr2=0x7fb358100900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:32.992 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.987+0000 7fb35e8d2700 1 --2- 172.21.15.121:0/3320280405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3581004e0 0x7fb358100900 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00aad0 tx=0x7fb34c00ade0 comp rx=0 tx=0).stop 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.987+0000 7fb35e8d2700 1 -- 172.21.15.121:0/3320280405 shutdown_connections 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.987+0000 7fb35e8d2700 1 --2- 172.21.15.121:0/3320280405 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb358102080 0x7fb358108930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.987+0000 7fb35e8d2700 1 --2- 172.21.15.121:0/3320280405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3581004e0 0x7fb358100900 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.987+0000 7fb35e8d2700 1 --2- 172.21.15.121:0/3320280405 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3581016e0 0x7fb358101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.987+0000 7fb35e8d2700 1 -- 172.21.15.121:0/3320280405 >> 172.21.15.121:0/3320280405 conn(0x7fb3580fba80 msgr2=0x7fb3580fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.987+0000 7fb35e8d2700 1 -- 172.21.15.121:0/3320280405 shutdown_connections 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.987+0000 7fb35e8d2700 1 -- 172.21.15.121:0/3320280405 wait complete. 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb35e8d2700 1 Processor -- start 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb35e8d2700 1 -- start start 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb35e8d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3581004e0 0x7fb358198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb35e8d2700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3581016e0 0x7fb3581985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb35e8d2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb358102080 0x7fb35819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:32.993 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb35e8d2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35810b8b0 con 0x7fb358102080 2024-03-26T06:32:32.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb35e8d2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35810b730 con 0x7fb3581004e0 2024-03-26T06:32:32.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb35e8d2700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35810ba30 con 0x7fb3581016e0 2024-03-26T06:32:32.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb357fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3581004e0 0x7fb358198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:32.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb357fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3581004e0 0x7fb358198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:40470/0 (socket says 172.21.15.121:40470) 2024-03-26T06:32:32.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb357fff700 1 -- 172.21.15.121:0/1999445687 learned_addr learned my addr 172.21.15.121:0/1999445687 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-03-26T06:32:32.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb3577fe700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3581016e0 0x7fb3581985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:32.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.988+0000 7fb35ce6f700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb358102080 0x7fb35819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:32.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb357fff700 1 -- 172.21.15.121:0/1999445687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3581016e0 msgr2=0x7fb3581985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:32.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb357fff700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3581016e0 0x7fb3581985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:32.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb357fff700 1 -- 172.21.15.121:0/1999445687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb358102080 msgr2=0x7fb35819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:32.994 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb357fff700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb358102080 0x7fb35819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:32.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb357fff700 1 -- 172.21.15.121:0/1999445687 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35819fcf0 con 0x7fb3581004e0 2024-03-26T06:32:32.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb35ce6f700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb358102080 0x7fb35819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:32:32.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb357fff700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3581004e0 0x7fb358198090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c690 tx=0x7fb34c004220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:32:32.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb3557fa700 1 -- 172.21.15.121:0/1999445687 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c023020 con 0x7fb3581004e0 2024-03-26T06:32:32.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb35e8d2700 1 -- 172.21.15.121:0/1999445687 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb35819ff80 con 0x7fb3581004e0 2024-03-26T06:32:32.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb3557fa700 1 -- 172.21.15.121:0/1999445687 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb34c003890 con 0x7fb3581004e0 2024-03-26T06:32:32.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb3557fa700 1 -- 172.21.15.121:0/1999445687 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34c016680 con 0x7fb3581004e0 2024-03-26T06:32:32.995 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.989+0000 7fb35e8d2700 1 -- 172.21.15.121:0/1999445687 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3581a0450 con 0x7fb3581004e0 2024-03-26T06:32:32.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.990+0000 7fb35e8d2700 1 -- 172.21.15.121:0/1999445687 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb35802d830 con 0x7fb3581004e0 2024-03-26T06:32:32.996 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.991+0000 7fb3557fa700 1 -- 172.21.15.121:0/1999445687 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb34c026020 con 0x7fb3581004e0 2024-03-26T06:32:32.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.992+0000 7fb3557fa700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb3400a2890 0x7fb3400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:32.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.992+0000 7fb3557fa700 1 -- 172.21.15.121:0/1999445687 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fb34c0cf1e0 con 0x7fb3581004e0 2024-03-26T06:32:32.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.992+0000 7fb3577fe700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb3400a2890 0x7fb3400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:32.997 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.992+0000 7fb3577fe700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb3400a2890 0x7fb3400a4d50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb348009a10 tx=0x7fb348009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:32:32.998 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:32.996+0000 7fb3557fa700 1 -- 172.21.15.121:0/1999445687 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb34c02d030 con 0x7fb3581004e0 2024-03-26T06:32:33.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:32 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi038 2024-03-26T06:32:33.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:32 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi038 to 5697M 2024-03-26T06:32:33.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:32 smithi038 ceph-mon[32265]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T06:32:33.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:32 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi038 2024-03-26T06:32:33.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:32 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi038 to 5697M 2024-03-26T06:32:33.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:32 smithi170 ceph-mon[36912]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T06:32:33.202 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:33.200+0000 7fb35e8d2700 1 -- 172.21.15.121:0/1999445687 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb358062220 con 0x7fb3400a2890 2024-03-26T06:32:34.056 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:33 smithi121 ceph-mon[36349]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:34.056 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:32:34.057 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:32:34.057 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:34.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:33 smithi038 ceph-mon[32265]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:34.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:32:34.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:32:34.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:34.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:33 smithi170 ceph-mon[36912]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:34.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:32:34.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:32:34.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:34.937 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:34 smithi121 ceph-mon[36349]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:32:35.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:34 smithi038 ceph-mon[32265]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:32:35.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:34 smithi170 ceph-mon[36912]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:32:36.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:35 smithi038 ceph-mon[32265]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:36.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:35 smithi038 ceph-mon[32265]: from='client.? 172.21.15.121:0/3736199660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11318d82-7c42-4e41-81b9-ecf84b65cd61"}]: dispatch 2024-03-26T06:32:36.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:35 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11318d82-7c42-4e41-81b9-ecf84b65cd61"}]: dispatch 2024-03-26T06:32:36.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:35 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11318d82-7c42-4e41-81b9-ecf84b65cd61"}]': finished 2024-03-26T06:32:36.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:35 smithi038 ceph-mon[32265]: osdmap e49: 8 total, 7 up, 8 in 2024-03-26T06:32:36.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:35 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:36.094 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:35 smithi121 ceph-mon[36349]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:36.094 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:35 smithi121 ceph-mon[36349]: from='client.? 172.21.15.121:0/3736199660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11318d82-7c42-4e41-81b9-ecf84b65cd61"}]: dispatch 2024-03-26T06:32:36.094 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:35 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11318d82-7c42-4e41-81b9-ecf84b65cd61"}]: dispatch 2024-03-26T06:32:36.094 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:35 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11318d82-7c42-4e41-81b9-ecf84b65cd61"}]': finished 2024-03-26T06:32:36.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:35 smithi121 ceph-mon[36349]: osdmap e49: 8 total, 7 up, 8 in 2024-03-26T06:32:36.095 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:35 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:36.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:35 smithi170 ceph-mon[36912]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:36.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:35 smithi170 ceph-mon[36912]: from='client.? 172.21.15.121:0/3736199660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11318d82-7c42-4e41-81b9-ecf84b65cd61"}]: dispatch 2024-03-26T06:32:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:35 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "11318d82-7c42-4e41-81b9-ecf84b65cd61"}]: dispatch 2024-03-26T06:32:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:35 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "11318d82-7c42-4e41-81b9-ecf84b65cd61"}]': finished 2024-03-26T06:32:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:35 smithi170 ceph-mon[36912]: osdmap e49: 8 total, 7 up, 8 in 2024-03-26T06:32:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:35 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:37.068 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:36 smithi170 ceph-mon[36912]: from='client.? 172.21.15.121:0/4074325625' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:32:37.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:36 smithi038 ceph-mon[32265]: from='client.? 172.21.15.121:0/4074325625' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:32:37.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:36 smithi121 ceph-mon[36349]: from='client.? 172.21.15.121:0/4074325625' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:32:38.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:37 smithi038 ceph-mon[32265]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:38.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:37 smithi121 ceph-mon[36349]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:38.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:37 smithi170 ceph-mon[36912]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:39.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:39 smithi121 ceph-mon[36349]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:40.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:39 smithi038 ceph-mon[32265]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:40.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:39 smithi170 ceph-mon[36912]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:41.975 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:41 smithi121 ceph-mon[36349]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:42.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:41 smithi038 ceph-mon[32265]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:42.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:41 smithi170 ceph-mon[36912]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:44.086 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:43 smithi121 ceph-mon[36349]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:44.086 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T06:32:44.086 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:44.086 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:43 smithi121 ceph-mon[36349]: Deploying daemon osd.7 on smithi121 2024-03-26T06:32:44.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:43 smithi170 ceph-mon[36912]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:44.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T06:32:44.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:44.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:43 smithi170 ceph-mon[36912]: Deploying daemon osd.7 on smithi121 2024-03-26T06:32:44.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:43 smithi038 ceph-mon[32265]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:44.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T06:32:44.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:44.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:43 smithi038 ceph-mon[32265]: Deploying daemon osd.7 on smithi121 2024-03-26T06:32:46.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:45 smithi121 ceph-mon[36349]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:46.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:45 smithi170 ceph-mon[36912]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:46.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:45 smithi038 ceph-mon[32265]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:47.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:47 smithi121 ceph-mon[36349]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:48.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:47 smithi170 ceph-mon[36912]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:48.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:47 smithi038 ceph-mon[32265]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:49.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:49 smithi038 ceph-mon[32265]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:49.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:49.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:49.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:49.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:49 smithi121 ceph-mon[36349]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:49.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:49.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:49.619 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:49.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:49 smithi170 ceph-mon[36912]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:49.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:49.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:49.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.075 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 7 on host 'smithi121' 2024-03-26T06:32:52.076 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.072+0000 7fb3557fa700 1 -- 172.21.15.121:0/1999445687 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb358062220 con 0x7fb3400a2890 2024-03-26T06:32:52.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 -- 172.21.15.121:0/1999445687 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb3400a2890 msgr2=0x7fb3400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:52.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb3400a2890 0x7fb3400a4d50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb348009a10 tx=0x7fb348009190 comp rx=0 tx=0).stop 2024-03-26T06:32:52.077 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 -- 172.21.15.121:0/1999445687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3581004e0 msgr2=0x7fb358198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:52.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3581004e0 0x7fb358198090 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb34c00c690 tx=0x7fb34c004220 comp rx=0 tx=0).stop 2024-03-26T06:32:52.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 -- 172.21.15.121:0/1999445687 shutdown_connections 2024-03-26T06:32:52.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb358102080 0x7fb35819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:52.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb3400a2890 0x7fb3400a4d50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:52.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb3581004e0 0x7fb358198090 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:52.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 --2- 172.21.15.121:0/1999445687 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb3581016e0 0x7fb3581985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:52.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 -- 172.21.15.121:0/1999445687 >> 172.21.15.121:0/1999445687 conn(0x7fb3580fba80 msgr2=0x7fb3581049e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:32:52.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 -- 172.21.15.121:0/1999445687 shutdown_connections 2024-03-26T06:32:52.078 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:32:52.075+0000 7fb35e8d2700 1 -- 172.21.15.121:0/1999445687 wait complete. 2024-03-26T06:32:52.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:52 smithi121 ceph-mon[36349]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:52.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:52.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:52.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:52.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:52 smithi121 ceph-mon[36349]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T06:32:52.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:52 smithi170 ceph-mon[36912]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:52.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:52.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:52.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:52.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:52 smithi170 ceph-mon[36912]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T06:32:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:52 smithi038 ceph-mon[32265]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:52.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:32:52.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:52 smithi038 ceph-mon[32265]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T06:32:52.725 DEBUG:teuthology.orchestra.run.smithi121:osd.7> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T06:32:52.728 INFO:tasks.cephadm:Deploying osd.8 on smithi170 with /dev/vg_nvme/lv_2... 2024-03-26T06:32:52.728 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T06:32:52.974 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:32:54.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:54 smithi121 ceph-mon[36349]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:54.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:54 smithi121 ceph-mon[36349]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T06:32:54.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:54 smithi121 ceph-mon[36349]: osdmap e50: 8 total, 7 up, 8 in 2024-03-26T06:32:54.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:54 smithi121 ceph-mon[36349]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:32:54.259 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:54 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:54.320 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:54 smithi170 ceph-mon[36912]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:54.320 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:54 smithi170 ceph-mon[36912]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T06:32:54.320 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:54 smithi170 ceph-mon[36912]: osdmap e50: 8 total, 7 up, 8 in 2024-03-26T06:32:54.320 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:54 smithi170 ceph-mon[36912]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:32:54.320 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:54 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:54.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:54 smithi038 ceph-mon[32265]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:54.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:54 smithi038 ceph-mon[32265]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T06:32:54.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:54 smithi038 ceph-mon[32265]: osdmap e50: 8 total, 7 up, 8 in 2024-03-26T06:32:54.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:54 smithi038 ceph-mon[32265]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:32:54.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:54 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:55.002 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T06:32:55.003 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T06:32:55.003 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-26T06:32:55.003 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-26T06:32:55.003 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00869136 s, 1.2 GB/s 2024-03-26T06:32:55.003 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 2024-03-26T06:32:55.003 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-26T06:32:55.302 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:55 smithi170 ceph-mon[36912]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:32:55.302 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:55 smithi170 ceph-mon[36912]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T06:32:55.302 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:55 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:55.303 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:55 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:55.303 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:55 smithi170 ceph-mon[36912]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' 2024-03-26T06:32:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:55 smithi038 ceph-mon[32265]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:32:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:55 smithi038 ceph-mon[32265]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T06:32:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:55 smithi038 ceph-mon[32265]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' 2024-03-26T06:32:55.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:55 smithi121 ceph-mon[36349]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:32:55.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:55 smithi121 ceph-mon[36349]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T06:32:55.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:55 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:55.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:55 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:55.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:55 smithi121 ceph-mon[36349]: from='osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891]' entity='osd.7' 2024-03-26T06:32:55.395 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:32:54 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:32:54.919+0000 7f01e46c7700 -1 osd.7 0 waiting for initial osdmap 2024-03-26T06:32:55.395 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:32:54 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:32:54.934+0000 7f01dccb3700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:32:55.609 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_2 2024-03-26T06:32:55.849 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:32:56.223 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:32:56.223 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:32:56.223 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:56.223 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891] boot 2024-03-26T06:32:56.223 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: osdmap e52: 8 total, 8 up, 8 in 2024-03-26T06:32:56.223 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:56.223 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.224 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.224 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:56.224 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:56.224 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:56.224 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.224 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:56.224 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:56.224 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.224 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:56 smithi170 ceph-mon[36912]: osdmap e53: 8 total, 8 up, 8 in 2024-03-26T06:32:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:32:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:32:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891] boot 2024-03-26T06:32:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: osdmap e52: 8 total, 8 up, 8 in 2024-03-26T06:32:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:56 smithi038 ceph-mon[32265]: osdmap e53: 8 total, 8 up, 8 in 2024-03-26T06:32:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:32:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:32:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-26T06:32:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: osd.7 [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891] boot 2024-03-26T06:32:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: osdmap e52: 8 total, 8 up, 8 in 2024-03-26T06:32:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:32:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:56.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:56.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:32:56.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:56.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:32:56.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:32:56.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:56 smithi121 ceph-mon[36349]: osdmap e53: 8 total, 8 up, 8 in 2024-03-26T06:32:57.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:57 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi121 2024-03-26T06:32:57.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:57 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi121 to 5693M 2024-03-26T06:32:57.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:57 smithi170 ceph-mon[36912]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T06:32:57.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:57 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi121 2024-03-26T06:32:57.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:57 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi121 to 5693M 2024-03-26T06:32:57.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:57 smithi038 ceph-mon[32265]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T06:32:57.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:57 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi121 2024-03-26T06:32:57.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:57 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi121 to 5693M 2024-03-26T06:32:57.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:57 smithi121 ceph-mon[36349]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T06:32:57.421 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.417+0000 7f9c60b9f700 1 -- 172.21.15.170:0/1891178681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c5c1028f0 msgr2=0x7f9c5c1091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:57.421 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.417+0000 7f9c60b9f700 1 --2- 172.21.15.170:0/1891178681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c5c1028f0 0x7f9c5c1091a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400b0a0 tx=0x7f9c5400b3b0 comp rx=0 tx=0).stop 2024-03-26T06:32:57.421 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.417+0000 7f9c60b9f700 1 -- 172.21.15.170:0/1891178681 shutdown_connections 2024-03-26T06:32:57.421 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.417+0000 7f9c60b9f700 1 --2- 172.21.15.170:0/1891178681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c5c1028f0 0x7f9c5c1091a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:57.421 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.417+0000 7f9c60b9f700 1 --2- 172.21.15.170:0/1891178681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c5c100d50 0x7f9c5c101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.417+0000 7f9c60b9f700 1 --2- 172.21.15.170:0/1891178681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c5c101f50 0x7f9c5c1023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.417+0000 7f9c60b9f700 1 -- 172.21.15.170:0/1891178681 >> 172.21.15.170:0/1891178681 conn(0x7f9c5c0fc2d0 msgr2=0x7f9c5c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.417+0000 7f9c60b9f700 1 -- 172.21.15.170:0/1891178681 shutdown_connections 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.417+0000 7f9c60b9f700 1 -- 172.21.15.170:0/1891178681 wait complete. 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.418+0000 7f9c60b9f700 1 Processor -- start 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.418+0000 7f9c60b9f700 1 -- start start 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.418+0000 7f9c60b9f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c5c100d50 0x7f9c5c06b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.418+0000 7f9c60b9f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c5c101f50 0x7f9c5c19ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.418+0000 7f9c60b9f700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c5c1028f0 0x7f9c5c19f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.418+0000 7f9c60b9f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c10c920 con 0x7f9c5c101f50 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.418+0000 7f9c60b9f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c10c7a0 con 0x7f9c5c100d50 2024-03-26T06:32:57.422 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.418+0000 7f9c60b9f700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c5c10caa0 con 0x7f9c5c1028f0 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.418+0000 7f9c5ad9d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c5c1028f0 0x7f9c5c19f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c5ad9d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c5c1028f0 0x7f9c5c19f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:35162/0 (socket says 172.21.15.170:35162) 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c5ad9d700 1 -- 172.21.15.170:0/684117425 learned_addr learned my addr 172.21.15.170:0/684117425 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c59d9b700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c5c101f50 0x7f9c5c19ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c5a59c700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c5c100d50 0x7f9c5c06b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c5ad9d700 1 -- 172.21.15.170:0/684117425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c5c100d50 msgr2=0x7f9c5c06b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c5ad9d700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c5c100d50 0x7f9c5c06b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c5ad9d700 1 -- 172.21.15.170:0/684117425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c5c101f50 msgr2=0x7f9c5c19ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c5ad9d700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c5c101f50 0x7f9c5c19ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c5ad9d700 1 -- 172.21.15.170:0/684117425 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c19fbd0 con 0x7f9c5c1028f0 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c59d9b700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c5c101f50 0x7f9c5c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c5ad9d700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c5c1028f0 0x7f9c5c19f300 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400b070 tx=0x7f9c5400be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c4b7fe700 1 -- 172.21.15.170:0/684117425 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c54021070 con 0x7f9c5c1028f0 2024-03-26T06:32:57.423 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c4b7fe700 1 -- 172.21.15.170:0/684117425 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c54005520 con 0x7f9c5c1028f0 2024-03-26T06:32:57.424 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c60b9f700 1 -- 172.21.15.170:0/684117425 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c5c19fe60 con 0x7f9c5c1028f0 2024-03-26T06:32:57.424 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c4b7fe700 1 -- 172.21.15.170:0/684117425 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5401b900 con 0x7f9c5c1028f0 2024-03-26T06:32:57.424 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.419+0000 7f9c60b9f700 1 -- 172.21.15.170:0/684117425 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5c1a0250 con 0x7f9c5c1028f0 2024-03-26T06:32:57.425 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.420+0000 7f9c60b9f700 1 -- 172.21.15.170:0/684117425 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c5c02d830 con 0x7f9c5c1028f0 2024-03-26T06:32:57.425 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.422+0000 7f9c4b7fe700 1 -- 172.21.15.170:0/684117425 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9c54024070 con 0x7f9c5c1028f0 2024-03-26T06:32:57.425 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.422+0000 7f9c4b7fe700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9c440a2890 0x7f9c440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:32:57.425 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.422+0000 7f9c4b7fe700 1 -- 172.21.15.170:0/684117425 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f9c5400f020 con 0x7f9c5c1028f0 2024-03-26T06:32:57.425 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.423+0000 7f9c5a59c700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9c440a2890 0x7f9c440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:32:57.425 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.423+0000 7f9c5a59c700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9c440a2890 0x7f9c440a4d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c005950 tx=0x7f9c4c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:32:57.428 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.426+0000 7f9c4b7fe700 1 -- 172.21.15.170:0/684117425 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9c5402b020 con 0x7f9c5c1028f0 2024-03-26T06:32:57.629 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:32:57.627+0000 7f9c60b9f700 1 -- 172.21.15.170:0/684117425 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c5c062220 con 0x7f9c440a2890 2024-03-26T06:32:58.330 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:58 smithi170 ceph-mon[36912]: pgmap v178: 1 pgs: 1 unknown; 577 KiB data, 392 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:32:58.331 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:58 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:32:58.331 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:58 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:32:58.331 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:58 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:58.331 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:58 smithi170 ceph-mon[36912]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T06:32:58.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:58 smithi038 ceph-mon[32265]: pgmap v178: 1 pgs: 1 unknown; 577 KiB data, 392 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:32:58.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:32:58.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:32:58.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:58.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:58 smithi038 ceph-mon[32265]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T06:32:58.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:58 smithi121 ceph-mon[36349]: pgmap v178: 1 pgs: 1 unknown; 577 KiB data, 392 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:32:58.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:58 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:32:58.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:58 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:32:58.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:58 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:32:58.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:58 smithi121 ceph-mon[36349]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T06:32:59.354 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:59 smithi170 ceph-mon[36912]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:32:59.354 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:32:59 smithi170 ceph-mon[36912]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T06:32:59.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:59 smithi121 ceph-mon[36349]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:32:59.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:32:59 smithi121 ceph-mon[36349]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T06:32:59.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:59 smithi038 ceph-mon[32265]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:32:59.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:32:59 smithi038 ceph-mon[32265]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T06:33:00.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:00 smithi121 ceph-mon[36349]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 392 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:00.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:00 smithi170 ceph-mon[36912]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 392 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:00 smithi038 ceph-mon[32265]: pgmap v181: 1 pgs: 1 unknown; 577 KiB data, 392 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:01.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:01 smithi121 ceph-mon[36349]: from='client.? 172.21.15.170:0/578436336' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0"}]: dispatch 2024-03-26T06:33:01.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:01 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0"}]: dispatch 2024-03-26T06:33:01.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:01 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0"}]': finished 2024-03-26T06:33:01.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:01 smithi121 ceph-mon[36349]: osdmap e57: 9 total, 8 up, 9 in 2024-03-26T06:33:01.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:01 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:01.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:01 smithi121 ceph-mon[36349]: from='client.? 172.21.15.170:0/2360000829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:33:01.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:01 smithi170 ceph-mon[36912]: from='client.? 172.21.15.170:0/578436336' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0"}]: dispatch 2024-03-26T06:33:01.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:01 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0"}]: dispatch 2024-03-26T06:33:01.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:01 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0"}]': finished 2024-03-26T06:33:01.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:01 smithi170 ceph-mon[36912]: osdmap e57: 9 total, 8 up, 9 in 2024-03-26T06:33:01.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:01 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:01.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:01 smithi170 ceph-mon[36912]: from='client.? 172.21.15.170:0/2360000829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:33:01.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:01 smithi038 ceph-mon[32265]: from='client.? 172.21.15.170:0/578436336' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0"}]: dispatch 2024-03-26T06:33:01.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:01 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0"}]: dispatch 2024-03-26T06:33:01.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:01 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0"}]': finished 2024-03-26T06:33:01.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:01 smithi038 ceph-mon[32265]: osdmap e57: 9 total, 8 up, 9 in 2024-03-26T06:33:01.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:01.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:01 smithi038 ceph-mon[32265]: from='client.? 172.21.15.170:0/2360000829' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:33:02.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:02 smithi121 ceph-mon[36349]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:02.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:02 smithi170 ceph-mon[36912]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:02.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:02 smithi038 ceph-mon[32265]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:04.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:04 smithi170 ceph-mon[36912]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:04.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:04 smithi121 ceph-mon[36349]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:04.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:04 smithi038 ceph-mon[32265]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:06.221 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:06 smithi170 ceph-mon[36912]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:06.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:06 smithi038 ceph-mon[32265]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:06.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:06 smithi121 ceph-mon[36349]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:08.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:08 smithi170 ceph-mon[36912]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:08.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-26T06:33:08.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:08.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:08 smithi121 ceph-mon[36349]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:08.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-26T06:33:08.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:08.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:08 smithi038 ceph-mon[32265]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:08.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-26T06:33:08.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:09.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:09 smithi121 ceph-mon[36349]: Deploying daemon osd.8 on smithi170 2024-03-26T06:33:09.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:09 smithi170 ceph-mon[36912]: Deploying daemon osd.8 on smithi170 2024-03-26T06:33:09.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:09 smithi038 ceph-mon[32265]: Deploying daemon osd.8 on smithi170 2024-03-26T06:33:10.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:10 smithi170 ceph-mon[36912]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:10.401 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:10 smithi121 ceph-mon[36349]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:10 smithi038 ceph-mon[32265]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:12.217 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:12 smithi170 ceph-mon[36912]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:12.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:12 smithi038 ceph-mon[32265]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:12.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:12 smithi121 ceph-mon[36349]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:14.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:13 smithi038 ceph-mon[32265]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:14.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:14.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:14.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:14.137 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:13 smithi170 ceph-mon[36912]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:14.138 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:14.138 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:14.138 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:14.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:13 smithi121 ceph-mon[36349]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:16.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:15 smithi170 ceph-mon[36912]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:16.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:16.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:16.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:16.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:16.029 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:15 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:16.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:15 smithi038 ceph-mon[32265]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:16.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:16.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:16.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:16.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:16.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:15 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:16.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:15 smithi121 ceph-mon[36349]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:16.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:15 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:16.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:15 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:16.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:15 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:16.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:15 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:16.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:15 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:16.352 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.350+0000 7f9c4b7fe700 1 -- 172.21.15.170:0/684117425 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f9c5c062220 con 0x7f9c440a2890 2024-03-26T06:33:16.353 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 8 on host 'smithi170' 2024-03-26T06:33:16.354 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 -- 172.21.15.170:0/684117425 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9c440a2890 msgr2=0x7f9c440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:33:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9c440a2890 0x7f9c440a4d50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c005950 tx=0x7f9c4c0058e0 comp rx=0 tx=0).stop 2024-03-26T06:33:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 -- 172.21.15.170:0/684117425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c5c1028f0 msgr2=0x7f9c5c19f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:33:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c5c1028f0 0x7f9c5c19f300 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9c5400b070 tx=0x7f9c5400be20 comp rx=0 tx=0).stop 2024-03-26T06:33:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 -- 172.21.15.170:0/684117425 shutdown_connections 2024-03-26T06:33:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9c5c1028f0 0x7f9c5c19f300 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9c440a2890 0x7f9c440a4d50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9c5c100d50 0x7f9c5c06b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 --2- 172.21.15.170:0/684117425 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c5c101f50 0x7f9c5c19ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 -- 172.21.15.170:0/684117425 >> 172.21.15.170:0/684117425 conn(0x7f9c5c0fc2d0 msgr2=0x7f9c5c1051f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:33:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 -- 172.21.15.170:0/684117425 shutdown_connections 2024-03-26T06:33:16.355 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:33:16.352+0000 7f9c60b9f700 1 -- 172.21.15.170:0/684117425 wait complete. 2024-03-26T06:33:16.964 DEBUG:teuthology.orchestra.run.smithi170:osd.8> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.8.service 2024-03-26T06:33:16.966 INFO:tasks.cephadm:Deploying osd.9 on smithi038 with /dev/vg_nvme/lv_1... 2024-03-26T06:33:16.966 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T06:33:17.217 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:33:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:17 smithi038 ceph-mon[32265]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:17 smithi038 ceph-mon[32265]: from='osd.8 [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-26T06:33:17.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:17 smithi038 ceph-mon[32265]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-26T06:33:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:17 smithi121 ceph-mon[36349]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:17 smithi121 ceph-mon[36349]: from='osd.8 [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-26T06:33:17.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:17 smithi121 ceph-mon[36349]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-26T06:33:17.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:17 smithi170 ceph-mon[36912]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:17.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:17.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:17.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:17 smithi170 ceph-mon[36912]: from='osd.8 [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-26T06:33:17.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:17 smithi170 ceph-mon[36912]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-26T06:33:18.606 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:18 smithi038 ceph-mon[32265]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-26T06:33:18.606 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:18 smithi038 ceph-mon[32265]: from='osd.8 [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:33:18.606 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:18 smithi038 ceph-mon[32265]: osdmap e58: 9 total, 8 up, 9 in 2024-03-26T06:33:18.606 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:18 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:18.606 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:18 smithi038 ceph-mon[32265]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:33:18.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:18 smithi121 ceph-mon[36349]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-26T06:33:18.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:18 smithi121 ceph-mon[36349]: from='osd.8 [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:33:18.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:18 smithi121 ceph-mon[36349]: osdmap e58: 9 total, 8 up, 9 in 2024-03-26T06:33:18.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:18 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:18.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:18 smithi121 ceph-mon[36349]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:33:18.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:18 smithi170 ceph-mon[36912]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-26T06:33:18.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:18 smithi170 ceph-mon[36912]: from='osd.8 [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:33:18.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:18 smithi170 ceph-mon[36912]: osdmap e58: 9 total, 8 up, 9 in 2024-03-26T06:33:18.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:18 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:18.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:18 smithi170 ceph-mon[36912]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:33:19.259 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T06:33:19.260 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T06:33:19.260 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2024-03-26T06:33:19.260 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2024-03-26T06:33:19.260 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00721815 s, 1.5 GB/s 2024-03-26T06:33:19.260 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2024-03-26T06:33:19.295 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:33:18 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:33:18.990+0000 7f8c1d2b7700 -1 osd.8 0 waiting for initial osdmap 2024-03-26T06:33:19.295 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:33:19 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:33:19.006+0000 7f8c158a3700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:33:19.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:19 smithi038 ceph-mon[32265]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:19.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:19 smithi038 ceph-mon[32265]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:33:19.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:19 smithi038 ceph-mon[32265]: osdmap e59: 9 total, 8 up, 9 in 2024-03-26T06:33:19.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:19.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:19.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:19 smithi038 ceph-mon[32265]: from='osd.8 ' entity='osd.8' 2024-03-26T06:33:19.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:19 smithi038 ceph-mon[32265]: osd.8 [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971] boot 2024-03-26T06:33:19.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:19 smithi038 ceph-mon[32265]: osdmap e60: 9 total, 9 up, 9 in 2024-03-26T06:33:19.567 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:19.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:19 smithi121 ceph-mon[36349]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:19.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:19 smithi121 ceph-mon[36349]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:33:19.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:19 smithi121 ceph-mon[36349]: osdmap e59: 9 total, 8 up, 9 in 2024-03-26T06:33:19.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:19.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:19.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:19 smithi121 ceph-mon[36349]: from='osd.8 ' entity='osd.8' 2024-03-26T06:33:19.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:19 smithi121 ceph-mon[36349]: osd.8 [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971] boot 2024-03-26T06:33:19.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:19 smithi121 ceph-mon[36349]: osdmap e60: 9 total, 9 up, 9 in 2024-03-26T06:33:19.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:19.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:19 smithi170 ceph-mon[36912]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-26T06:33:19.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:19 smithi170 ceph-mon[36912]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:33:19.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:19 smithi170 ceph-mon[36912]: osdmap e59: 9 total, 8 up, 9 in 2024-03-26T06:33:19.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:19.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:19.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:19 smithi170 ceph-mon[36912]: from='osd.8 ' entity='osd.8' 2024-03-26T06:33:19.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:19 smithi170 ceph-mon[36912]: osd.8 [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971] boot 2024-03-26T06:33:19.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:19 smithi170 ceph-mon[36912]: osdmap e60: 9 total, 9 up, 9 in 2024-03-26T06:33:19.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:33:20.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi038:vg_nvme/lv_1 2024-03-26T06:33:20.395 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:33:20.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:33:20.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:33:20.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi170 2024-03-26T06:33:20.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:20.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:20.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:20.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi170 to 5693M 2024-03-26T06:33:20.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:20.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:20.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:20 smithi038 ceph-mon[32265]: osdmap e61: 9 total, 9 up, 9 in 2024-03-26T06:33:20.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:33:20.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:33:20.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi170 2024-03-26T06:33:20.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:20.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:20.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:20.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi170 to 5693M 2024-03-26T06:33:20.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:20.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:20.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:20 smithi121 ceph-mon[36349]: osdmap e61: 9 total, 9 up, 9 in 2024-03-26T06:33:20.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:33:20.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:33:20.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi170 2024-03-26T06:33:20.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi170 to 5693M 2024-03-26T06:33:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:20 smithi170 ceph-mon[36912]: osdmap e61: 9 total, 9 up, 9 in 2024-03-26T06:33:22.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.089+0000 7fa870c6d700 1 -- 172.21.15.38:0/3438079266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa86c102f70 msgr2=0x7fa86c1033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:33:22.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.089+0000 7fa870c6d700 1 --2- 172.21.15.38:0/3438079266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa86c102f70 0x7fa86c1033f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa860009990 tx=0x7fa860009ca0 comp rx=0 tx=0).stop 2024-03-26T06:33:22.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.089+0000 7fa870c6d700 1 -- 172.21.15.38:0/3438079266 shutdown_connections 2024-03-26T06:33:22.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.089+0000 7fa870c6d700 1 --2- 172.21.15.38:0/3438079266 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa86c103930 0x7fa86c10a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:22.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.089+0000 7fa870c6d700 1 --2- 172.21.15.38:0/3438079266 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa86c101e10 0x7fa86c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:22.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.089+0000 7fa870c6d700 1 --2- 172.21.15.38:0/3438079266 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa86c102f70 0x7fa86c1033f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:22.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.089+0000 7fa870c6d700 1 -- 172.21.15.38:0/3438079266 >> 172.21.15.38:0/3438079266 conn(0x7fa86c0fd3f0 msgr2=0x7fa86c0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:33:22.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.089+0000 7fa870c6d700 1 -- 172.21.15.38:0/3438079266 shutdown_connections 2024-03-26T06:33:22.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.089+0000 7fa870c6d700 1 -- 172.21.15.38:0/3438079266 wait complete. 2024-03-26T06:33:22.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.090+0000 7fa870c6d700 1 Processor -- start 2024-03-26T06:33:22.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.090+0000 7fa870c6d700 1 -- start start 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.090+0000 7fa870c6d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa86c101e10 0x7fa86c197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.090+0000 7fa870c6d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa86c102f70 0x7fa86c1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.090+0000 7fa870c6d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa86c103930 0x7fa86c19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.090+0000 7fa870c6d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c10da30 con 0x7fa86c101e10 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.090+0000 7fa870c6d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c10d8b0 con 0x7fa86c103930 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.090+0000 7fa870c6d700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86c10dbb0 con 0x7fa86c102f70 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.090+0000 7fa86a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa86c101e10 0x7fa86c197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.090+0000 7fa86a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa86c101e10 0x7fa86c197f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51482/0 (socket says 172.21.15.38:51482) 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.090+0000 7fa86a59c700 1 -- 172.21.15.38:0/491249971 learned_addr learned my addr 172.21.15.38:0/491249971 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa86ad9d700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa86c103930 0x7fa86c19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa869d9b700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa86c102f70 0x7fa86c1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:33:22.095 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa86a59c700 1 -- 172.21.15.38:0/491249971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa86c102f70 msgr2=0x7fa86c1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:33:22.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa86a59c700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa86c102f70 0x7fa86c1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:22.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa86a59c700 1 -- 172.21.15.38:0/491249971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa86c103930 msgr2=0x7fa86c19f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:33:22.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa86a59c700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa86c103930 0x7fa86c19f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:22.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa86a59c700 1 -- 172.21.15.38:0/491249971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c19fbc0 con 0x7fa86c101e10 2024-03-26T06:33:22.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa86ad9d700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa86c103930 0x7fa86c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:33:22.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa86a59c700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa86c101e10 0x7fa86c197f60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00c9e0 tx=0x7fa85c00cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:33:22.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa85b7fe700 1 -- 172.21.15.38:0/491249971 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c007ac0 con 0x7fa86c101e10 2024-03-26T06:33:22.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa870c6d700 1 -- 172.21.15.38:0/491249971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86c19feb0 con 0x7fa86c101e10 2024-03-26T06:33:22.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa85b7fe700 1 -- 172.21.15.38:0/491249971 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa85c007c60 con 0x7fa86c101e10 2024-03-26T06:33:22.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa85b7fe700 1 -- 172.21.15.38:0/491249971 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c0186f0 con 0x7fa86c101e10 2024-03-26T06:33:22.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.091+0000 7fa870c6d700 1 -- 172.21.15.38:0/491249971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86c1a0310 con 0x7fa86c101e10 2024-03-26T06:33:22.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.092+0000 7fa870c6d700 1 -- 172.21.15.38:0/491249971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86c068aa0 con 0x7fa86c101e10 2024-03-26T06:33:22.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.093+0000 7fa85b7fe700 1 -- 172.21.15.38:0/491249971 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa85c0188f0 con 0x7fa86c101e10 2024-03-26T06:33:22.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.094+0000 7fa85b7fe700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa8540a2990 0x7fa8540a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:33:22.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.094+0000 7fa869d9b700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa8540a2990 0x7fa8540a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:33:22.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.094+0000 7fa85b7fe700 1 -- 172.21.15.38:0/491249971 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5858+0+0 (secure 0 0 0) 0x7fa85c0ce150 con 0x7fa86c101e10 2024-03-26T06:33:22.098 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.094+0000 7fa869d9b700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa8540a2990 0x7fa8540a4e50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa86000b5c0 tx=0x7fa8600038f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:33:22.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.098+0000 7fa85b7fe700 1 -- 172.21.15.38:0/491249971 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa85c014030 con 0x7fa86c101e10 2024-03-26T06:33:22.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:22.313+0000 7fa870c6d700 1 -- 172.21.15.38:0/491249971 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa86c062220 con 0x7fa8540a2990 2024-03-26T06:33:22.315 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:22 smithi038 ceph-mon[32265]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:22.315 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:22 smithi038 ceph-mon[32265]: osdmap e62: 9 total, 9 up, 9 in 2024-03-26T06:33:22.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:22 smithi121 ceph-mon[36349]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:22.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:22 smithi121 ceph-mon[36349]: osdmap e62: 9 total, 9 up, 9 in 2024-03-26T06:33:22.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:22 smithi170 ceph-mon[36912]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:22.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:22 smithi170 ceph-mon[36912]: osdmap e62: 9 total, 9 up, 9 in 2024-03-26T06:33:23.183 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:33:23.183 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:33:23.183 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:23.183 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:23 smithi038 ceph-mon[32265]: osdmap e63: 9 total, 9 up, 9 in 2024-03-26T06:33:23.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:33:23.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:33:23.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:23.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:23 smithi121 ceph-mon[36349]: osdmap e63: 9 total, 9 up, 9 in 2024-03-26T06:33:23.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:33:23.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:33:23.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:23.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:23 smithi170 ceph-mon[36912]: osdmap e63: 9 total, 9 up, 9 in 2024-03-26T06:33:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:24 smithi121 ceph-mon[36349]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:24 smithi121 ceph-mon[36349]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:33:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:24 smithi121 ceph-mon[36349]: osdmap e64: 9 total, 9 up, 9 in 2024-03-26T06:33:24.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:24 smithi170 ceph-mon[36912]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:24.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:24 smithi170 ceph-mon[36912]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:33:24.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:24 smithi170 ceph-mon[36912]: osdmap e64: 9 total, 9 up, 9 in 2024-03-26T06:33:24.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:24 smithi038 ceph-mon[32265]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:24.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:24 smithi038 ceph-mon[32265]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:33:24.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:24 smithi038 ceph-mon[32265]: osdmap e64: 9 total, 9 up, 9 in 2024-03-26T06:33:25.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:25 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2004364056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fa2735f-8c3c-41bd-8de1-2681f659088c"}]: dispatch 2024-03-26T06:33:25.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:25 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2004364056' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fa2735f-8c3c-41bd-8de1-2681f659088c"}]': finished 2024-03-26T06:33:25.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:25 smithi121 ceph-mon[36349]: osdmap e65: 10 total, 9 up, 10 in 2024-03-26T06:33:25.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:25 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:25 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2004364056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fa2735f-8c3c-41bd-8de1-2681f659088c"}]: dispatch 2024-03-26T06:33:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:25 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2004364056' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fa2735f-8c3c-41bd-8de1-2681f659088c"}]': finished 2024-03-26T06:33:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:25 smithi170 ceph-mon[36912]: osdmap e65: 10 total, 9 up, 10 in 2024-03-26T06:33:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:25.484 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:25 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2004364056' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0fa2735f-8c3c-41bd-8de1-2681f659088c"}]: dispatch 2024-03-26T06:33:25.484 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:25 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2004364056' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0fa2735f-8c3c-41bd-8de1-2681f659088c"}]': finished 2024-03-26T06:33:25.485 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:25 smithi038 ceph-mon[32265]: osdmap e65: 10 total, 9 up, 10 in 2024-03-26T06:33:25.485 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:26 smithi038 ceph-mon[32265]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:26 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3541483567' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:33:26.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:26 smithi121 ceph-mon[36349]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:26.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:26 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3541483567' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:33:26.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:26 smithi170 ceph-mon[36912]: pgmap v203: 1 pgs: 1 remapped+peering; 577 KiB data, 672 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:26.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:26 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3541483567' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:33:28.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:27 smithi038 ceph-mon[32265]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-26T06:33:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:27 smithi121 ceph-mon[36349]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-26T06:33:28.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:27 smithi170 ceph-mon[36912]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-26T06:33:30.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:29 smithi038 ceph-mon[32265]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T06:33:30.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:29 smithi121 ceph-mon[36349]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T06:33:30.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:29 smithi170 ceph-mon[36912]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T06:33:32.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:31 smithi038 ceph-mon[32265]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T06:33:32.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:31 smithi121 ceph-mon[36349]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T06:33:32.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:31 smithi170 ceph-mon[36912]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T06:33:34.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:33 smithi038 ceph-mon[32265]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T06:33:34.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-26T06:33:34.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:34.144 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:33 smithi121 ceph-mon[36349]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T06:33:34.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-26T06:33:34.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:34.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:33 smithi170 ceph-mon[36912]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T06:33:34.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-26T06:33:34.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:34.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:34 smithi038 ceph-mon[32265]: Deploying daemon osd.9 on smithi038 2024-03-26T06:33:35.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:34 smithi121 ceph-mon[36349]: Deploying daemon osd.9 on smithi038 2024-03-26T06:33:35.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:34 smithi170 ceph-mon[36912]: Deploying daemon osd.9 on smithi038 2024-03-26T06:33:35.975 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:35 smithi038 ceph-mon[32265]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T06:33:36.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:35 smithi121 ceph-mon[36349]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T06:33:36.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:35 smithi170 ceph-mon[36912]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T06:33:37.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:37 smithi038 ceph-mon[32265]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-26T06:33:38.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:37 smithi121 ceph-mon[36349]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-26T06:33:38.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:37 smithi170 ceph-mon[36912]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 51 KiB/s, 0 objects/s recovering 2024-03-26T06:33:40.061 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:39 smithi038 ceph-mon[32265]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:40.061 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:40.061 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:40.061 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:39 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:40.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:39 smithi121 ceph-mon[36349]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:40.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:40.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:40.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:39 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:40.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:39 smithi170 ceph-mon[36912]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:40.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:40.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:40.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:39 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:41.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:41 smithi038 ceph-mon[32265]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:42.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:41 smithi121 ceph-mon[36349]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:42.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:41 smithi170 ceph-mon[36912]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:43.223 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.221+0000 7fa85b7fe700 1 -- 172.21.15.38:0/491249971 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa86c062220 con 0x7fa8540a2990 2024-03-26T06:33:43.224 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 9 on host 'smithi038' 2024-03-26T06:33:43.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.223+0000 7fa870c6d700 1 -- 172.21.15.38:0/491249971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa8540a2990 msgr2=0x7fa8540a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:33:43.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.223+0000 7fa870c6d700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa8540a2990 0x7fa8540a4e50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fa86000b5c0 tx=0x7fa8600038f0 comp rx=0 tx=0).stop 2024-03-26T06:33:43.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.223+0000 7fa870c6d700 1 -- 172.21.15.38:0/491249971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa86c101e10 msgr2=0x7fa86c197f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:33:43.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.223+0000 7fa870c6d700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa86c101e10 0x7fa86c197f60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa85c00c9e0 tx=0x7fa85c00cda0 comp rx=0 tx=0).stop 2024-03-26T06:33:43.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.223+0000 7fa870c6d700 1 -- 172.21.15.38:0/491249971 shutdown_connections 2024-03-26T06:33:43.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.223+0000 7fa870c6d700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa8540a2990 0x7fa8540a4e50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:43.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.223+0000 7fa870c6d700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa86c103930 0x7fa86c19f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:43.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.223+0000 7fa870c6d700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa86c101e10 0x7fa86c197f60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:43.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.223+0000 7fa870c6d700 1 --2- 172.21.15.38:0/491249971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa86c102f70 0x7fa86c1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:43.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.223+0000 7fa870c6d700 1 -- 172.21.15.38:0/491249971 >> 172.21.15.38:0/491249971 conn(0x7fa86c0fd3f0 msgr2=0x7fa86c1062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:33:43.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.224+0000 7fa870c6d700 1 -- 172.21.15.38:0/491249971 shutdown_connections 2024-03-26T06:33:43.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:33:43.224+0000 7fa870c6d700 1 -- 172.21.15.38:0/491249971 wait complete. 2024-03-26T06:33:43.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:43 smithi038 ceph-mon[32265]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:43.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:43.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:43.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:43 smithi038 ceph-mon[32265]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-26T06:33:43.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:43 smithi121 ceph-mon[36349]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:43 smithi121 ceph-mon[36349]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-26T06:33:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:43.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:43 smithi170 ceph-mon[36912]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:43.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:43 smithi170 ceph-mon[36912]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-26T06:33:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:33:43.843 DEBUG:teuthology.orchestra.run.smithi038:osd.9> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.9.service 2024-03-26T06:33:43.845 INFO:tasks.cephadm:Deploying osd.10 on smithi121 with /dev/vg_nvme/lv_1... 2024-03-26T06:33:43.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T06:33:44.092 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:33:45.465 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:45 smithi121 ceph-mon[36349]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-26T06:33:45.465 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:45 smithi121 ceph-mon[36349]: osdmap e66: 10 total, 9 up, 10 in 2024-03-26T06:33:45.465 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:45 smithi121 ceph-mon[36349]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:33:45.465 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:45 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:45.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:45 smithi038 ceph-mon[32265]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-26T06:33:45.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:45 smithi038 ceph-mon[32265]: osdmap e66: 10 total, 9 up, 10 in 2024-03-26T06:33:45.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:45 smithi038 ceph-mon[32265]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:33:45.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:45 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:45.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:33:45 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:33:45.188+0000 7fa3da0c6700 -1 osd.9 0 waiting for initial osdmap 2024-03-26T06:33:45.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:33:45 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:33:45.202+0000 7fa3d66ba700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:33:45.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:45 smithi170 ceph-mon[36912]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-26T06:33:45.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:45 smithi170 ceph-mon[36912]: osdmap e66: 10 total, 9 up, 10 in 2024-03-26T06:33:45.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:45 smithi170 ceph-mon[36912]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2024-03-26T06:33:45.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:45 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:46.016 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T06:33:46.016 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T06:33:46.016 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10+0 records in 2024-03-26T06:33:46.016 INFO:teuthology.orchestra.run.smithi121.stderr:10+0 records out 2024-03-26T06:33:46.016 INFO:teuthology.orchestra.run.smithi121.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011818 s, 887 MB/s 2024-03-26T06:33:46.016 INFO:teuthology.orchestra.run.smithi121.stderr:--> Zapping successful for: 2024-03-26T06:33:46.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:46 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:33:46.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:46 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:33:46.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:46 smithi038 ceph-mon[32265]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:46.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:46 smithi038 ceph-mon[32265]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:33:46.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:46 smithi038 ceph-mon[32265]: osdmap e67: 10 total, 9 up, 10 in 2024-03-26T06:33:46.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:46 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:46.308 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:46 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:46.478 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:46 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:33:46.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:46 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:33:46.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:46 smithi121 ceph-mon[36349]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:46.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:46 smithi121 ceph-mon[36349]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:33:46.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:46 smithi121 ceph-mon[36349]: osdmap e67: 10 total, 9 up, 10 in 2024-03-26T06:33:46.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:46 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:46.479 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:46 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:46.638 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi121:vg_nvme/lv_1 2024-03-26T06:33:46.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:46 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:33:46.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:46 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:33:46.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:46 smithi170 ceph-mon[36912]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-26T06:33:46.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:46 smithi170 ceph-mon[36912]: from='osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2024-03-26T06:33:46.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:46 smithi170 ceph-mon[36912]: osdmap e67: 10 total, 9 up, 10 in 2024-03-26T06:33:46.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:46 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:46.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:46 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:46.894 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:33:47.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510] boot 2024-03-26T06:33:47.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: osdmap e68: 10 total, 10 up, 10 in 2024-03-26T06:33:47.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:47.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:47.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:47.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:47.289 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:47.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:47.290 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:47 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:47.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510] boot 2024-03-26T06:33:47.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: osdmap e68: 10 total, 10 up, 10 in 2024-03-26T06:33:47.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:47.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:47.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:47.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:47.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:47.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:47.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:47 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:47.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: osd.9 [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510] boot 2024-03-26T06:33:47.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: osdmap e68: 10 total, 10 up, 10 in 2024-03-26T06:33:47.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:33:47.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:47.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:47.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:33:47.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:47.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:47.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:33:47.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:47 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:33:48.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:48 smithi121 ceph-mon[36349]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:48.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:48 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi038 2024-03-26T06:33:48.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:48 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi038 to 4273M 2024-03-26T06:33:48.388 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:48 smithi121 ceph-mon[36349]: osdmap e69: 10 total, 10 up, 10 in 2024-03-26T06:33:48.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.404+0000 7fe221933700 1 -- 172.21.15.121:0/1845857734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe21c107ba0 msgr2=0x7fe21c109f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:33:48.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.404+0000 7fe221933700 1 --2- 172.21.15.121:0/1845857734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe21c107ba0 0x7fe21c109f90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe214009990 tx=0x7fe214009ca0 comp rx=0 tx=0).stop 2024-03-26T06:33:48.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.404+0000 7fe221933700 1 -- 172.21.15.121:0/1845857734 shutdown_connections 2024-03-26T06:33:48.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.404+0000 7fe221933700 1 --2- 172.21.15.121:0/1845857734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe21c107ba0 0x7fe21c109f90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:48.409 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.404+0000 7fe221933700 1 --2- 172.21.15.121:0/1845857734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe21c102940 0x7fe21c104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:48.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.404+0000 7fe221933700 1 --2- 172.21.15.121:0/1845857734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe21c105270 0x7fe21c107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:48.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.404+0000 7fe221933700 1 -- 172.21.15.121:0/1845857734 >> 172.21.15.121:0/1845857734 conn(0x7fe21c0fc2b0 msgr2=0x7fe21c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:33:48.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.404+0000 7fe221933700 1 -- 172.21.15.121:0/1845857734 shutdown_connections 2024-03-26T06:33:48.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.404+0000 7fe221933700 1 -- 172.21.15.121:0/1845857734 wait complete. 2024-03-26T06:33:48.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe221933700 1 Processor -- start 2024-03-26T06:33:48.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe221933700 1 -- start start 2024-03-26T06:33:48.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe221933700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe21c102940 0x7fe21c198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:33:48.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe221933700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe21c105270 0x7fe21c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:33:48.410 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe221933700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe21c107ba0 0x7fe21c19f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:33:48.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe221933700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c10d9e0 con 0x7fe21c107ba0 2024-03-26T06:33:48.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe221933700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c10d860 con 0x7fe21c105270 2024-03-26T06:33:48.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe221933700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c10db60 con 0x7fe21c102940 2024-03-26T06:33:48.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe21a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe21c105270 0x7fe21c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:33:48.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe21a7fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe21c105270 0x7fe21c198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:48978/0 (socket says 172.21.15.121:48978) 2024-03-26T06:33:48.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe21a7fc700 1 -- 172.21.15.121:0/578125621 learned_addr learned my addr 172.21.15.121:0/578125621 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-03-26T06:33:48.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe21affd700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe21c102940 0x7fe21c198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:33:48.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.405+0000 7fe21b7fe700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe21c107ba0 0x7fe21c19f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:33:48.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe21a7fc700 1 -- 172.21.15.121:0/578125621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe21c102940 msgr2=0x7fe21c198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:33:48.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe21a7fc700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe21c102940 0x7fe21c198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:48.411 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe21a7fc700 1 -- 172.21.15.121:0/578125621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe21c107ba0 msgr2=0x7fe21c19f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:33:48.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe21a7fc700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe21c107ba0 0x7fe21c19f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:33:48.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe21a7fc700 1 -- 172.21.15.121:0/578125621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c19fc80 con 0x7fe21c105270 2024-03-26T06:33:48.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe21affd700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe21c102940 0x7fe21c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:33:48.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe21a7fc700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe21c105270 0x7fe21c198560 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe20400e9a0 tx=0x7fe20400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:33:48.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe220931700 1 -- 172.21.15.121:0/578125621 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe20400ca10 con 0x7fe21c105270 2024-03-26T06:33:48.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe220931700 1 -- 172.21.15.121:0/578125621 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe20400cbb0 con 0x7fe21c105270 2024-03-26T06:33:48.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe221933700 1 -- 172.21.15.121:0/578125621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c19ff70 con 0x7fe21c105270 2024-03-26T06:33:48.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe220931700 1 -- 172.21.15.121:0/578125621 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe204018710 con 0x7fe21c105270 2024-03-26T06:33:48.412 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.406+0000 7fe221933700 1 -- 172.21.15.121:0/578125621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c1a0420 con 0x7fe21c105270 2024-03-26T06:33:48.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.407+0000 7fe221933700 1 -- 172.21.15.121:0/578125621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c068aa0 con 0x7fe21c105270 2024-03-26T06:33:48.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.409+0000 7fe220931700 1 -- 172.21.15.121:0/578125621 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe2040188b0 con 0x7fe21c105270 2024-03-26T06:33:48.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.409+0000 7fe220931700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe2080a2890 0x7fe2080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:33:48.413 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.410+0000 7fe220931700 1 -- 172.21.15.121:0/578125621 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fe2040ce590 con 0x7fe21c105270 2024-03-26T06:33:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.410+0000 7fe21affd700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe2080a2890 0x7fe2080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:33:48.414 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.410+0000 7fe21affd700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe2080a2890 0x7fe2080a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe20c003e30 tx=0x7fe20c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:33:48.415 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.413+0000 7fe220931700 1 -- 172.21.15.121:0/578125621 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe20409be30 con 0x7fe21c105270 2024-03-26T06:33:48.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:48 smithi038 ceph-mon[32265]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:48.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:48 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi038 2024-03-26T06:33:48.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:48 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi038 to 4273M 2024-03-26T06:33:48.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:48 smithi038 ceph-mon[32265]: osdmap e69: 10 total, 10 up, 10 in 2024-03-26T06:33:48.616 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:33:48.613+0000 7fe221933700 1 -- 172.21.15.121:0/578125621 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe21c062220 con 0x7fe2080a2890 2024-03-26T06:33:48.666 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:48 smithi170 ceph-mon[36912]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:48.666 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:48 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi038 2024-03-26T06:33:48.666 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:48 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi038 to 4273M 2024-03-26T06:33:48.666 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:48 smithi170 ceph-mon[36912]: osdmap e69: 10 total, 10 up, 10 in 2024-03-26T06:33:49.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:49 smithi121 ceph-mon[36349]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:49.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:49 smithi121 ceph-mon[36349]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:33:49.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:33:49.524 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:33:49.525 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:49.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:49 smithi038 ceph-mon[32265]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:49.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:49 smithi038 ceph-mon[32265]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:33:49.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:33:49.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:33:49.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:49.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:49 smithi170 ceph-mon[36912]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:49.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:49 smithi170 ceph-mon[36912]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi121:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:33:49.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:33:49.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:33:49.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:51.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:51 smithi038 ceph-mon[32265]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:51.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:51 smithi121 ceph-mon[36349]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:51.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:51 smithi170 ceph-mon[36912]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:52 smithi038 ceph-mon[32265]: from='client.? 172.21.15.121:0/404649979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa25661a-1c05-40d1-a17c-94d274eb01f8"}]: dispatch 2024-03-26T06:33:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:52 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa25661a-1c05-40d1-a17c-94d274eb01f8"}]: dispatch 2024-03-26T06:33:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:52 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa25661a-1c05-40d1-a17c-94d274eb01f8"}]': finished 2024-03-26T06:33:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:52 smithi038 ceph-mon[32265]: osdmap e70: 11 total, 10 up, 11 in 2024-03-26T06:33:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:52 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:33:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:52 smithi038 ceph-mon[32265]: from='client.? 172.21.15.121:0/3363308629' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:33:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:52 smithi121 ceph-mon[36349]: from='client.? 172.21.15.121:0/404649979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa25661a-1c05-40d1-a17c-94d274eb01f8"}]: dispatch 2024-03-26T06:33:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:52 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa25661a-1c05-40d1-a17c-94d274eb01f8"}]: dispatch 2024-03-26T06:33:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:52 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa25661a-1c05-40d1-a17c-94d274eb01f8"}]': finished 2024-03-26T06:33:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:52 smithi121 ceph-mon[36349]: osdmap e70: 11 total, 10 up, 11 in 2024-03-26T06:33:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:52 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:33:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:52 smithi121 ceph-mon[36349]: from='client.? 172.21.15.121:0/3363308629' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:33:52.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:52 smithi170 ceph-mon[36912]: from='client.? 172.21.15.121:0/404649979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa25661a-1c05-40d1-a17c-94d274eb01f8"}]: dispatch 2024-03-26T06:33:52.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:52 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa25661a-1c05-40d1-a17c-94d274eb01f8"}]: dispatch 2024-03-26T06:33:52.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:52 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa25661a-1c05-40d1-a17c-94d274eb01f8"}]': finished 2024-03-26T06:33:52.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:52 smithi170 ceph-mon[36912]: osdmap e70: 11 total, 10 up, 11 in 2024-03-26T06:33:52.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:52 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:33:52.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:52 smithi170 ceph-mon[36912]: from='client.? 172.21.15.121:0/3363308629' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:33:53.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:53 smithi121 ceph-mon[36349]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:53.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:53 smithi170 ceph-mon[36912]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:53.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:53 smithi038 ceph-mon[32265]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:55.635 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:55 smithi038 ceph-mon[32265]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:55.636 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:55 smithi121 ceph-mon[36349]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:55.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:55 smithi170 ceph-mon[36912]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:57 smithi121 ceph-mon[36349]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:57.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:57 smithi170 ceph-mon[36912]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:57.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:57 smithi038 ceph-mon[32265]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:59.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:59 smithi121 ceph-mon[36349]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:59.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:59 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-26T06:33:59.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:59 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:59.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:33:59 smithi121 ceph-mon[36349]: Deploying daemon osd.10 on smithi121 2024-03-26T06:33:59.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:59 smithi170 ceph-mon[36912]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:59.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:59 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-26T06:33:59.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:59 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:59.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:33:59 smithi170 ceph-mon[36912]: Deploying daemon osd.10 on smithi121 2024-03-26T06:33:59.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:59 smithi038 ceph-mon[32265]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:33:59.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:59 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-26T06:33:59.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:59 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:33:59.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:33:59 smithi038 ceph-mon[32265]: Deploying daemon osd.10 on smithi121 2024-03-26T06:34:01.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:01 smithi121 ceph-mon[36349]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:01.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:01 smithi170 ceph-mon[36912]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:01.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:01 smithi038 ceph-mon[32265]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:03.505 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:03 smithi121 ceph-mon[36349]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:03.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:03 smithi170 ceph-mon[36912]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:03 smithi038 ceph-mon[32265]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:05.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:05 smithi038 ceph-mon[32265]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:05.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:05.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:05.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:05 smithi121 ceph-mon[36349]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:05.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:05 smithi170 ceph-mon[36912]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:05.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:05.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:05.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:07.408 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:07 smithi121 ceph-mon[36349]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:07.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:07 smithi038 ceph-mon[32265]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:07.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:07 smithi170 ceph-mon[36912]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:07.782 INFO:teuthology.orchestra.run.smithi121.stdout:Created osd(s) 10 on host 'smithi121' 2024-03-26T06:34:07.782 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.779+0000 7fe220931700 1 -- 172.21.15.121:0/578125621 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fe21c062220 con 0x7fe2080a2890 2024-03-26T06:34:07.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 -- 172.21.15.121:0/578125621 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe2080a2890 msgr2=0x7fe2080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:07.783 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe2080a2890 0x7fe2080a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe20c003e30 tx=0x7fe20c00b040 comp rx=0 tx=0).stop 2024-03-26T06:34:07.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 -- 172.21.15.121:0/578125621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe21c105270 msgr2=0x7fe21c198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:07.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe21c105270 0x7fe21c198560 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe20400e9a0 tx=0x7fe20400ecb0 comp rx=0 tx=0).stop 2024-03-26T06:34:07.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 -- 172.21.15.121:0/578125621 shutdown_connections 2024-03-26T06:34:07.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe21c107ba0 0x7fe21c19f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:07.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe21c102940 0x7fe21c198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:07.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe2080a2890 0x7fe2080a4d50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:07.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 --2- 172.21.15.121:0/578125621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe21c105270 0x7fe21c198560 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:07.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 -- 172.21.15.121:0/578125621 >> 172.21.15.121:0/578125621 conn(0x7fe21c0fc2b0 msgr2=0x7fe21c1031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:07.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 -- 172.21.15.121:0/578125621 shutdown_connections 2024-03-26T06:34:07.784 INFO:teuthology.orchestra.run.smithi121.stderr:2024-03-26T06:34:07.781+0000 7fe221933700 1 -- 172.21.15.121:0/578125621 wait complete. 2024-03-26T06:34:08.366 DEBUG:teuthology.orchestra.run.smithi121:osd.10> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.10.service 2024-03-26T06:34:08.368 INFO:tasks.cephadm:Deploying osd.11 on smithi170 with /dev/vg_nvme/lv_1... 2024-03-26T06:34:08.369 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T06:34:08.614 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:34:09.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:09.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:09.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:08 smithi170 ceph-mon[36912]: from='osd.10 [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-26T06:34:09.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:08 smithi170 ceph-mon[36912]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-26T06:34:09.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:09.047 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:09.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:09.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:09.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:08 smithi038 ceph-mon[32265]: from='osd.10 [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-26T06:34:09.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:08 smithi038 ceph-mon[32265]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-26T06:34:09.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:09.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:09.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:09.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:09.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:08 smithi121 ceph-mon[36349]: from='osd.10 [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-26T06:34:09.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:08 smithi121 ceph-mon[36349]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-26T06:34:09.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:09.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:10.002 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:09 smithi170 ceph-mon[36912]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:10.002 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:09 smithi170 ceph-mon[36912]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-26T06:34:10.002 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:09 smithi170 ceph-mon[36912]: from='osd.10 [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:34:10.003 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:09 smithi170 ceph-mon[36912]: osdmap e71: 11 total, 10 up, 11 in 2024-03-26T06:34:10.003 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:09 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:10.003 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:09 smithi170 ceph-mon[36912]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:34:10.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:09 smithi038 ceph-mon[32265]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:10.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:09 smithi038 ceph-mon[32265]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-26T06:34:10.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:09 smithi038 ceph-mon[32265]: from='osd.10 [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:34:10.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:09 smithi038 ceph-mon[32265]: osdmap e71: 11 total, 10 up, 11 in 2024-03-26T06:34:10.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:09 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:10.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:09 smithi038 ceph-mon[32265]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:34:10.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:09 smithi121 ceph-mon[36349]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:10.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:09 smithi121 ceph-mon[36349]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-26T06:34:10.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:09 smithi121 ceph-mon[36349]: from='osd.10 [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:34:10.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:09 smithi121 ceph-mon[36349]: osdmap e71: 11 total, 10 up, 11 in 2024-03-26T06:34:10.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:09 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:10.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:09 smithi121 ceph-mon[36349]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]: dispatch 2024-03-26T06:34:10.146 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:34:09 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:34:09.782+0000 7f08b7be5700 -1 osd.10 0 waiting for initial osdmap 2024-03-26T06:34:10.146 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:34:09 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:34:09.802+0000 7f08b41d9700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:34:10.515 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T06:34:10.515 INFO:teuthology.orchestra.run.smithi170.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T06:34:10.515 INFO:teuthology.orchestra.run.smithi170.stderr: stderr: 10+0 records in 2024-03-26T06:34:10.515 INFO:teuthology.orchestra.run.smithi170.stderr:10+0 records out 2024-03-26T06:34:10.515 INFO:teuthology.orchestra.run.smithi170.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00851234 s, 1.2 GB/s 2024-03-26T06:34:10.515 INFO:teuthology.orchestra.run.smithi170.stderr:--> Zapping successful for: 2024-03-26T06:34:11.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:10 smithi038 ceph-mon[32265]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:34:11.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:10 smithi038 ceph-mon[32265]: osdmap e72: 11 total, 10 up, 11 in 2024-03-26T06:34:11.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:10 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:11.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:10 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:11.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:10 smithi121 ceph-mon[36349]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:34:11.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:10 smithi121 ceph-mon[36349]: osdmap e72: 11 total, 10 up, 11 in 2024-03-26T06:34:11.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:10 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:11.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:10 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:11.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:10 smithi170 ceph-mon[36912]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi121", "root=default"]}]': finished 2024-03-26T06:34:11.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:10 smithi170 ceph-mon[36912]: osdmap e72: 11 total, 10 up, 11 in 2024-03-26T06:34:11.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:10 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:11.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:10 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:11.251 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi170:vg_nvme/lv_1 2024-03-26T06:34:11.486 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:34:11.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:11 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:34:11.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:11 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:34:11.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:11 smithi121 ceph-mon[36349]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:11.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:11 smithi121 ceph-mon[36349]: osd.10 [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853] boot 2024-03-26T06:34:11.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:11 smithi121 ceph-mon[36349]: osdmap e73: 11 total, 11 up, 11 in 2024-03-26T06:34:11.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:11 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:11.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:11 smithi121 ceph-mon[36349]: osdmap e74: 11 total, 11 up, 11 in 2024-03-26T06:34:11.911 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:11 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:34:11.911 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:11 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:34:11.911 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:11 smithi170 ceph-mon[36912]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:11.911 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:11 smithi170 ceph-mon[36912]: osd.10 [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853] boot 2024-03-26T06:34:11.911 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:11 smithi170 ceph-mon[36912]: osdmap e73: 11 total, 11 up, 11 in 2024-03-26T06:34:11.911 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:11 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:11.911 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:11 smithi170 ceph-mon[36912]: osdmap e74: 11 total, 11 up, 11 in 2024-03-26T06:34:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:11 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:34:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:11 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:34:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:11 smithi038 ceph-mon[32265]: pgmap v234: 1 pgs: 1 peering; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-26T06:34:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:11 smithi038 ceph-mon[32265]: osd.10 [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853] boot 2024-03-26T06:34:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:11 smithi038 ceph-mon[32265]: osdmap e73: 11 total, 11 up, 11 in 2024-03-26T06:34:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:11 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:34:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:11 smithi038 ceph-mon[32265]: osdmap e74: 11 total, 11 up, 11 in 2024-03-26T06:34:13.140 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.133+0000 7ff634572700 1 -- 172.21.15.170:0/1259356146 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62c1028e0 msgr2=0x7ff62c109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:13.140 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.133+0000 7ff634572700 1 --2- 172.21.15.170:0/1259356146 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62c1028e0 0x7ff62c109190 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff62400b0a0 tx=0x7ff62400b3b0 comp rx=0 tx=0).stop 2024-03-26T06:34:13.140 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.133+0000 7ff634572700 1 -- 172.21.15.170:0/1259356146 shutdown_connections 2024-03-26T06:34:13.140 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.133+0000 7ff634572700 1 --2- 172.21.15.170:0/1259356146 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62c1028e0 0x7ff62c109190 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:13.140 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.133+0000 7ff634572700 1 --2- 172.21.15.170:0/1259356146 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff62c100d40 0x7ff62c101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:13.140 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.133+0000 7ff634572700 1 --2- 172.21.15.170:0/1259356146 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff62c101f40 0x7ff62c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:13.140 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.133+0000 7ff634572700 1 -- 172.21.15.170:0/1259356146 >> 172.21.15.170:0/1259356146 conn(0x7ff62c0fc300 msgr2=0x7ff62c0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:13.140 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.134+0000 7ff634572700 1 -- 172.21.15.170:0/1259356146 shutdown_connections 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.134+0000 7ff634572700 1 -- 172.21.15.170:0/1259356146 wait complete. 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.134+0000 7ff634572700 1 Processor -- start 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.134+0000 7ff634572700 1 -- start start 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.134+0000 7ff634572700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff62c100d40 0x7ff62c06b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff634572700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff62c101f40 0x7ff62c19ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff634572700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62c1028e0 0x7ff62c19f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff634572700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c10c9a0 con 0x7ff62c101f40 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff634572700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c10c820 con 0x7ff62c100d40 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff634572700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff62c10cb20 con 0x7ff62c1028e0 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff632b0f700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62c1028e0 0x7ff62c19f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff632b0f700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62c1028e0 0x7ff62c19f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:56098/0 (socket says 172.21.15.170:56098) 2024-03-26T06:34:13.141 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff632b0f700 1 -- 172.21.15.170:0/1982113180 learned_addr learned my addr 172.21.15.170:0/1982113180 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff63230e700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff62c100d40 0x7ff62c06b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff632b0f700 1 -- 172.21.15.170:0/1982113180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff62c100d40 msgr2=0x7ff62c06b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff632b0f700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff62c100d40 0x7ff62c06b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff632b0f700 1 -- 172.21.15.170:0/1982113180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff62c101f40 msgr2=0x7ff62c19ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff631b0d700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff62c101f40 0x7ff62c19ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff632b0f700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff62c101f40 0x7ff62c19ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff632b0f700 1 -- 172.21.15.170:0/1982113180 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62c19fd20 con 0x7ff62c1028e0 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff632b0f700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62c1028e0 0x7ff62c19f450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff62400b070 tx=0x7ff62400bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff6237fe700 1 -- 172.21.15.170:0/1982113180 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff624021020 con 0x7ff62c1028e0 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff6237fe700 1 -- 172.21.15.170:0/1982113180 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff624005520 con 0x7ff62c1028e0 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff634572700 1 -- 172.21.15.170:0/1982113180 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62c19ffb0 con 0x7ff62c1028e0 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff6237fe700 1 -- 172.21.15.170:0/1982113180 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62401ba00 con 0x7ff62c1028e0 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.135+0000 7ff634572700 1 -- 172.21.15.170:0/1982113180 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62c1a0490 con 0x7ff62c1028e0 2024-03-26T06:34:13.142 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.136+0000 7ff634572700 1 -- 172.21.15.170:0/1982113180 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62c068aa0 con 0x7ff62c1028e0 2024-03-26T06:34:13.143 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.137+0000 7ff6237fe700 1 -- 172.21.15.170:0/1982113180 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff624024020 con 0x7ff62c1028e0 2024-03-26T06:34:13.143 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.138+0000 7ff6237fe700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff6180a2910 0x7ff6180a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:13.143 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.138+0000 7ff6237fe700 1 -- 172.21.15.170:0/1982113180 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6694+0+0 (secure 0 0 0) 0x7ff6240cfbf0 con 0x7ff62c1028e0 2024-03-26T06:34:13.144 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.138+0000 7ff63230e700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff6180a2910 0x7ff6180a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:13.144 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.139+0000 7ff63230e700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff6180a2910 0x7ff6180a4dd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff628006fd0 tx=0x7ff628008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:13.144 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.141+0000 7ff6237fe700 1 -- 172.21.15.170:0/1982113180 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff62402b020 con 0x7ff62c1028e0 2024-03-26T06:34:13.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi121 2024-03-26T06:34:13.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi121 to 4270M 2024-03-26T06:34:13.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:13.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:13.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:13.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:13.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:13.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: osdmap e75: 11 total, 11 up, 11 in 2024-03-26T06:34:13.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:13 smithi170 ceph-mon[36912]: osdmap e76: 11 total, 11 up, 11 in 2024-03-26T06:34:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi121 2024-03-26T06:34:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi121 to 4270M 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: osdmap e75: 11 total, 11 up, 11 in 2024-03-26T06:34:13.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:13 smithi038 ceph-mon[32265]: osdmap e76: 11 total, 11 up, 11 in 2024-03-26T06:34:13.350 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:13.347+0000 7ff634572700 1 -- 172.21.15.170:0/1982113180 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff62c062220 con 0x7ff6180a2910 2024-03-26T06:34:13.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi121 2024-03-26T06:34:13.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:13.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi121 to 4270M 2024-03-26T06:34:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: osdmap e75: 11 total, 11 up, 11 in 2024-03-26T06:34:13.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:13 smithi121 ceph-mon[36349]: osdmap e76: 11 total, 11 up, 11 in 2024-03-26T06:34:14.238 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:14 smithi170 ceph-mon[36912]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:14.238 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:14 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:34:14.238 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:14 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:34:14.238 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:14 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:14.238 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:14 smithi170 ceph-mon[36912]: osdmap e77: 11 total, 11 up, 11 in 2024-03-26T06:34:14.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:14 smithi038 ceph-mon[32265]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:14.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:14 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:34:14.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:14 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:34:14.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:14 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:14.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:14 smithi038 ceph-mon[32265]: osdmap e77: 11 total, 11 up, 11 in 2024-03-26T06:34:14.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:14 smithi121 ceph-mon[36349]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:14.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:14 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T06:34:14.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:14 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T06:34:14.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:14 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:14.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:14 smithi121 ceph-mon[36349]: osdmap e77: 11 total, 11 up, 11 in 2024-03-26T06:34:15.280 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:15 smithi170 ceph-mon[36912]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:34:15.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:15 smithi038 ceph-mon[32265]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:34:15.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:15 smithi121 ceph-mon[36349]: from='client.24346 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi170:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:34:16.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:16 smithi038 ceph-mon[32265]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:16.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:16 smithi121 ceph-mon[36349]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:16.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:16 smithi170 ceph-mon[36912]: pgmap v241: 1 pgs: 1 peering; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:17.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:17 smithi038 ceph-mon[32265]: from='client.? 172.21.15.170:0/3236108973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c870b245-cb9a-4275-9b4f-6f86c049991a"}]: dispatch 2024-03-26T06:34:17.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:17 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c870b245-cb9a-4275-9b4f-6f86c049991a"}]: dispatch 2024-03-26T06:34:17.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:17 smithi038 ceph-mon[32265]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c870b245-cb9a-4275-9b4f-6f86c049991a"}]': finished 2024-03-26T06:34:17.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:17 smithi038 ceph-mon[32265]: osdmap e78: 12 total, 11 up, 12 in 2024-03-26T06:34:17.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:17.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:17 smithi038 ceph-mon[32265]: from='client.? 172.21.15.170:0/995145003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:34:17.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:17 smithi121 ceph-mon[36349]: from='client.? 172.21.15.170:0/3236108973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c870b245-cb9a-4275-9b4f-6f86c049991a"}]: dispatch 2024-03-26T06:34:17.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:17 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c870b245-cb9a-4275-9b4f-6f86c049991a"}]: dispatch 2024-03-26T06:34:17.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:17 smithi121 ceph-mon[36349]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c870b245-cb9a-4275-9b4f-6f86c049991a"}]': finished 2024-03-26T06:34:17.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:17 smithi121 ceph-mon[36349]: osdmap e78: 12 total, 11 up, 12 in 2024-03-26T06:34:17.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:17.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:17 smithi121 ceph-mon[36349]: from='client.? 172.21.15.170:0/995145003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:34:17.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:17 smithi170 ceph-mon[36912]: from='client.? 172.21.15.170:0/3236108973' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c870b245-cb9a-4275-9b4f-6f86c049991a"}]: dispatch 2024-03-26T06:34:17.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:17 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c870b245-cb9a-4275-9b4f-6f86c049991a"}]: dispatch 2024-03-26T06:34:17.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:17 smithi170 ceph-mon[36912]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c870b245-cb9a-4275-9b4f-6f86c049991a"}]': finished 2024-03-26T06:34:17.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:17 smithi170 ceph-mon[36912]: osdmap e78: 12 total, 11 up, 12 in 2024-03-26T06:34:17.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:17.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:17 smithi170 ceph-mon[36912]: from='client.? 172.21.15.170:0/995145003' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T06:34:18.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:18 smithi121 ceph-mon[36349]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:34:18.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:18 smithi170 ceph-mon[36912]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:34:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:18 smithi038 ceph-mon[32265]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T06:34:20.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:20 smithi121 ceph-mon[36349]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-26T06:34:20.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:20 smithi170 ceph-mon[36912]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-26T06:34:20.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:20 smithi038 ceph-mon[32265]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-03-26T06:34:22.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:22 smithi121 ceph-mon[36349]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-26T06:34:22.400 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:22 smithi170 ceph-mon[36912]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-26T06:34:22.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:22 smithi038 ceph-mon[32265]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-26T06:34:24.320 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:24 smithi170 ceph-mon[36912]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-26T06:34:24.320 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:24 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-26T06:34:24.320 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:24 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:24 smithi121 ceph-mon[36349]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-26T06:34:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:24 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-26T06:34:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:24 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:24.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:24 smithi038 ceph-mon[32265]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-26T06:34:24.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:24 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-26T06:34:24.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:24 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:25.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:25 smithi121 ceph-mon[36349]: Deploying daemon osd.11 on smithi170 2024-03-26T06:34:25.410 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:25 smithi170 ceph-mon[36912]: Deploying daemon osd.11 on smithi170 2024-03-26T06:34:25.484 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:25 smithi038 ceph-mon[32265]: Deploying daemon osd.11 on smithi170 2024-03-26T06:34:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:26 smithi121 ceph-mon[36349]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T06:34:26.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:26 smithi170 ceph-mon[36912]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T06:34:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:26 smithi038 ceph-mon[32265]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T06:34:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:28 smithi121 ceph-mon[36349]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:28.405 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:28 smithi170 ceph-mon[36912]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:28.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:28 smithi038 ceph-mon[32265]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:30.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:30 smithi170 ceph-mon[36912]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:30.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:30 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:30.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:30 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:30.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:30 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:30.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:30 smithi038 ceph-mon[32265]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:30.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:30 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:30.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:30 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:30.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:30 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:30.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:30 smithi121 ceph-mon[36349]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:30.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:30 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:30.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:30 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:30.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:30 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:32.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:32 smithi170 ceph-mon[36912]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:32.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:32 smithi038 ceph-mon[32265]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:32 smithi121 ceph-mon[36349]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:33.022 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.019+0000 7ff6237fe700 1 -- 172.21.15.170:0/1982113180 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7ff62c062220 con 0x7ff6180a2910 2024-03-26T06:34:33.022 INFO:teuthology.orchestra.run.smithi170.stdout:Created osd(s) 11 on host 'smithi170' 2024-03-26T06:34:33.024 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 -- 172.21.15.170:0/1982113180 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff6180a2910 msgr2=0x7ff6180a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:33.024 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff6180a2910 0x7ff6180a4dd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff628006fd0 tx=0x7ff628008040 comp rx=0 tx=0).stop 2024-03-26T06:34:33.024 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 -- 172.21.15.170:0/1982113180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62c1028e0 msgr2=0x7ff62c19f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:33.024 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62c1028e0 0x7ff62c19f450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff62400b070 tx=0x7ff62400bf10 comp rx=0 tx=0).stop 2024-03-26T06:34:33.024 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 -- 172.21.15.170:0/1982113180 shutdown_connections 2024-03-26T06:34:33.024 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff62c1028e0 0x7ff62c19f450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:33.024 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff6180a2910 0x7ff6180a4dd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:33.024 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff62c100d40 0x7ff62c06b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:33.024 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 --2- 172.21.15.170:0/1982113180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff62c101f40 0x7ff62c19ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:33.024 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 -- 172.21.15.170:0/1982113180 >> 172.21.15.170:0/1982113180 conn(0x7ff62c0fc300 msgr2=0x7ff62c1051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:33.025 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 -- 172.21.15.170:0/1982113180 shutdown_connections 2024-03-26T06:34:33.025 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:33.021+0000 7ff634572700 1 -- 172.21.15.170:0/1982113180 wait complete. 2024-03-26T06:34:33.323 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.323 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.323 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:33.323 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:33.323 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.323 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:33.323 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.324 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:33 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.324 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:33 smithi170 ceph-mon[36912]: from='osd.11 [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-26T06:34:33.324 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:33 smithi170 ceph-mon[36912]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-26T06:34:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:33 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:33 smithi038 ceph-mon[32265]: from='osd.11 [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-26T06:34:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:33 smithi038 ceph-mon[32265]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-26T06:34:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:33 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:33 smithi121 ceph-mon[36349]: from='osd.11 [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-26T06:34:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:33 smithi121 ceph-mon[36349]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-26T06:34:33.658 DEBUG:teuthology.orchestra.run.smithi170:osd.11> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.11.service 2024-03-26T06:34:33.660 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-26T06:34:33.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T06:34:33.912 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:34:34.284 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:34 smithi038 ceph-mon[32265]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:34.285 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:34 smithi038 ceph-mon[32265]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-26T06:34:34.285 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:34 smithi038 ceph-mon[32265]: from='osd.11 [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:34:34.285 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:34 smithi038 ceph-mon[32265]: osdmap e79: 12 total, 11 up, 12 in 2024-03-26T06:34:34.285 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:34 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:34.285 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:34 smithi038 ceph-mon[32265]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:34:34.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:34 smithi170 ceph-mon[36912]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:34.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:34 smithi170 ceph-mon[36912]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-26T06:34:34.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:34 smithi170 ceph-mon[36912]: from='osd.11 [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:34:34.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:34 smithi170 ceph-mon[36912]: osdmap e79: 12 total, 11 up, 12 in 2024-03-26T06:34:34.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:34 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:34.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:34 smithi170 ceph-mon[36912]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:34:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:34 smithi121 ceph-mon[36349]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:34 smithi121 ceph-mon[36349]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-26T06:34:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:34 smithi121 ceph-mon[36349]: from='osd.11 [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:34:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:34 smithi121 ceph-mon[36349]: osdmap e79: 12 total, 11 up, 12 in 2024-03-26T06:34:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:34 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:34 smithi121 ceph-mon[36349]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]: dispatch 2024-03-26T06:34:35.085 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:34:35 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:34:35.025+0000 7f1f21de8700 -1 osd.11 0 waiting for initial osdmap 2024-03-26T06:34:35.086 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:34:35 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:34:35.040+0000 7f1f1e3dc700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T06:34:35.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.536+0000 7f7592b5a700 1 -- 172.21.15.38:0/1548408550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f758c1028a0 msgr2=0x7f758c109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:35.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.536+0000 7f7592b5a700 1 --2- 172.21.15.38:0/1548408550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f758c1028a0 0x7f758c109150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f758400b0d0 tx=0x7f758400b3e0 comp rx=0 tx=0).stop 2024-03-26T06:34:35.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.536+0000 7f7592b5a700 1 -- 172.21.15.38:0/1548408550 shutdown_connections 2024-03-26T06:34:35.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.536+0000 7f7592b5a700 1 --2- 172.21.15.38:0/1548408550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f758c1028a0 0x7f758c109150 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:35.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.536+0000 7f7592b5a700 1 --2- 172.21.15.38:0/1548408550 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f758c100d00 0x7f758c101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:35.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.536+0000 7f7592b5a700 1 --2- 172.21.15.38:0/1548408550 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f758c101f00 0x7f758c102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:35.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.536+0000 7f7592b5a700 1 -- 172.21.15.38:0/1548408550 >> 172.21.15.38:0/1548408550 conn(0x7f758c0fc2c0 msgr2=0x7f758c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:35.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.536+0000 7f7592b5a700 1 -- 172.21.15.38:0/1548408550 shutdown_connections 2024-03-26T06:34:35.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.536+0000 7f7592b5a700 1 -- 172.21.15.38:0/1548408550 wait complete. 2024-03-26T06:34:35.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f7592b5a700 1 Processor -- start 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f7592b5a700 1 -- start start 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f7592b5a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f758c100d00 0x7f758c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f7592b5a700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f758c101f00 0x7f758c198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f7592b5a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f758c1028a0 0x7f758c19f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f7592b5a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c10c960 con 0x7f758c100d00 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f7592b5a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c10c7e0 con 0x7f758c1028a0 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f7592b5a700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758c10cae0 con 0x7f758c101f00 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f75908f6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f758c100d00 0x7f758c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f75908f6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f758c100d00 0x7f758c198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43076/0 (socket says 172.21.15.38:43076) 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f75908f6700 1 -- 172.21.15.38:0/1672164527 learned_addr learned my addr 172.21.15.38:0/1672164527 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f75910f7700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f758c1028a0 0x7f758c19f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:35.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f758bfff700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f758c101f00 0x7f758c198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f75910f7700 1 -- 172.21.15.38:0/1672164527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f758c101f00 msgr2=0x7f758c198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f75910f7700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f758c101f00 0x7f758c198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.537+0000 7f75910f7700 1 -- 172.21.15.38:0/1672164527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f758c100d00 msgr2=0x7f758c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.538+0000 7f75910f7700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f758c100d00 0x7f758c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.538+0000 7f75910f7700 1 -- 172.21.15.38:0/1672164527 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f758c19fc90 con 0x7f758c1028a0 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.538+0000 7f75908f6700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f758c100d00 0x7f758c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.538+0000 7f75910f7700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f758c1028a0 0x7f758c19f5f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f758400d010 tx=0x7f7584007a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.538+0000 7f7589ffb700 1 -- 172.21.15.38:0/1672164527 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7584021070 con 0x7f758c1028a0 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.538+0000 7f7592b5a700 1 -- 172.21.15.38:0/1672164527 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758c19ff20 con 0x7f758c1028a0 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.538+0000 7f7589ffb700 1 -- 172.21.15.38:0/1672164527 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f758401db30 con 0x7f758c1028a0 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.538+0000 7f7589ffb700 1 -- 172.21.15.38:0/1672164527 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758401bba0 con 0x7f758c1028a0 2024-03-26T06:34:35.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.538+0000 7f7592b5a700 1 -- 172.21.15.38:0/1672164527 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758c1a0380 con 0x7f758c1028a0 2024-03-26T06:34:35.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.539+0000 7f7592b5a700 1 -- 172.21.15.38:0/1672164527 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758c068aa0 con 0x7f758c1028a0 2024-03-26T06:34:35.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.540+0000 7f7589ffb700 1 -- 172.21.15.38:0/1672164527 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7584024070 con 0x7f758c1028a0 2024-03-26T06:34:35.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.541+0000 7f7589ffb700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f75780a2910 0x7f75780a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:35.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.541+0000 7f75908f6700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f75780a2910 0x7f75780a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:35.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.541+0000 7f7589ffb700 1 -- 172.21.15.38:0/1672164527 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f758400f020 con 0x7f758c1028a0 2024-03-26T06:34:35.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.541+0000 7f75908f6700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f75780a2910 0x7f75780a4dd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f757c009e90 tx=0x7f757c009240 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:35.547 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.545+0000 7f7589ffb700 1 -- 172.21.15.38:0/1672164527 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f758402b020 con 0x7f758c1028a0 2024-03-26T06:34:35.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.748+0000 7f7592b5a700 1 -- 172.21.15.38:0/1672164527 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f758c1a0880 con 0x7f758c1028a0 2024-03-26T06:34:35.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.748+0000 7f7589ffb700 1 -- 172.21.15.38:0/1672164527 <== mon.2 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7f758409de20 con 0x7f758c1028a0 2024-03-26T06:34:35.751 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:34:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 -- 172.21.15.38:0/1672164527 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f75780a2910 msgr2=0x7f75780a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f75780a2910 0x7f75780a4dd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f757c009e90 tx=0x7f757c009240 comp rx=0 tx=0).stop 2024-03-26T06:34:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 -- 172.21.15.38:0/1672164527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f758c1028a0 msgr2=0x7f758c19f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f758c1028a0 0x7f758c19f5f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f758400d010 tx=0x7f7584007a20 comp rx=0 tx=0).stop 2024-03-26T06:34:35.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 -- 172.21.15.38:0/1672164527 shutdown_connections 2024-03-26T06:34:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f75780a2910 0x7f75780a4dd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f758c1028a0 0x7f758c19f5f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f758c100d00 0x7f758c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 --2- 172.21.15.38:0/1672164527 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f758c101f00 0x7f758c198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 -- 172.21.15.38:0/1672164527 >> 172.21.15.38:0/1672164527 conn(0x7f758c0fc2c0 msgr2=0x7f758c0fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 -- 172.21.15.38:0/1672164527 shutdown_connections 2024-03-26T06:34:35.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:35.751+0000 7f7592b5a700 1 -- 172.21.15.38:0/1672164527 wait complete. 2024-03-26T06:34:36.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:36 smithi170 ceph-mon[36912]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:36 smithi170 ceph-mon[36912]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:34:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:36 smithi170 ceph-mon[36912]: osdmap e80: 12 total, 11 up, 12 in 2024-03-26T06:34:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:36 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:36 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:36 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1672164527' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T06:34:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:36 smithi038 ceph-mon[32265]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:36 smithi038 ceph-mon[32265]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:34:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:36 smithi038 ceph-mon[32265]: osdmap e80: 12 total, 11 up, 12 in 2024-03-26T06:34:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:36 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:36.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:36 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:36.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:36 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1672164527' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T06:34:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:36 smithi121 ceph-mon[36349]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:36 smithi121 ceph-mon[36349]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi170", "root=default"]}]': finished 2024-03-26T06:34:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:36 smithi121 ceph-mon[36349]: osdmap e80: 12 total, 11 up, 12 in 2024-03-26T06:34:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:36 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:36 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:36 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1672164527' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T06:34:36.561 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1711434850,"num_in_osds":12,"osd_in_since":1711434856,"num_remapped_pgs":0} 2024-03-26T06:34:37.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: purged_snaps scrub starts 2024-03-26T06:34:37.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: purged_snaps scrub ok 2024-03-26T06:34:37.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:37.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: osd.11 [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754] boot 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: osdmap e81: 12 total, 12 up, 12 in 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:37.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:37 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: purged_snaps scrub starts 2024-03-26T06:34:37.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: purged_snaps scrub ok 2024-03-26T06:34:37.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:37.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: osd.11 [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754] boot 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: osdmap e81: 12 total, 12 up, 12 in 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:37 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: purged_snaps scrub starts 2024-03-26T06:34:37.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: purged_snaps scrub ok 2024-03-26T06:34:37.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:37.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: osd.11 [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754] boot 2024-03-26T06:34:37.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: osdmap e81: 12 total, 12 up, 12 in 2024-03-26T06:34:37.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:34:37.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:34:37.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:37.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:37.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:37 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:37.562 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T06:34:37.814 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:34:38.237 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:38 smithi038 ceph-mon[32265]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:38.237 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:38 smithi038 ceph-mon[32265]: Detected new or changed devices on smithi170 2024-03-26T06:34:38.237 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:38 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi170 to 4270M 2024-03-26T06:34:38.237 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:38 smithi038 ceph-mon[32265]: osdmap e82: 12 total, 12 up, 12 in 2024-03-26T06:34:38.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:38 smithi121 ceph-mon[36349]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:38.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:38 smithi121 ceph-mon[36349]: Detected new or changed devices on smithi170 2024-03-26T06:34:38.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:38 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi170 to 4270M 2024-03-26T06:34:38.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:38 smithi121 ceph-mon[36349]: osdmap e82: 12 total, 12 up, 12 in 2024-03-26T06:34:38.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:38 smithi170 ceph-mon[36912]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-03-26T06:34:38.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:38 smithi170 ceph-mon[36912]: Detected new or changed devices on smithi170 2024-03-26T06:34:38.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:38 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi170 to 4270M 2024-03-26T06:34:38.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:38 smithi170 ceph-mon[36912]: osdmap e82: 12 total, 12 up, 12 in 2024-03-26T06:34:39.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.479+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/3871990310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc100d10 msgr2=0x7fcdbc101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:39.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.479+0000 7fcdc0ad7700 1 --2- 172.21.15.38:0/3871990310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc100d10 0x7fcdbc101130 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fcda40099c0 tx=0x7fcda4009cd0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.479+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/3871990310 shutdown_connections 2024-03-26T06:34:39.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.479+0000 7fcdc0ad7700 1 --2- 172.21.15.38:0/3871990310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdbc1028b0 0x7fcdbc109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.479+0000 7fcdc0ad7700 1 --2- 172.21.15.38:0/3871990310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc100d10 0x7fcdbc101130 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.479+0000 7fcdc0ad7700 1 --2- 172.21.15.38:0/3871990310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdbc101f10 0x7fcdbc102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.479+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/3871990310 >> 172.21.15.38:0/3871990310 conn(0x7fcdbc0fc2b0 msgr2=0x7fcdbc0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:39.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.479+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/3871990310 shutdown_connections 2024-03-26T06:34:39.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.479+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/3871990310 wait complete. 2024-03-26T06:34:39.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.480+0000 7fcdc0ad7700 1 Processor -- start 2024-03-26T06:34:39.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.480+0000 7fcdc0ad7700 1 -- start start 2024-03-26T06:34:39.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdc0ad7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdbc100d10 0x7fcdbc198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:39.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdc0ad7700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdbc101f10 0x7fcdbc198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:39.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdc0ad7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1028b0 0x7fcdbc19f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:39.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdc0ad7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc10c8d0 con 0x7fcdbc1028b0 2024-03-26T06:34:39.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdc0ad7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc10c750 con 0x7fcdbc100d10 2024-03-26T06:34:39.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdc0ad7700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdbc10ca50 con 0x7fcdbc101f10 2024-03-26T06:34:39.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdbad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1028b0 0x7fcdbc19f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:39.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdbad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1028b0 0x7fcdbc19f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43096/0 (socket says 172.21.15.38:43096) 2024-03-26T06:34:39.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdbad9d700 1 -- 172.21.15.38:0/359796640 learned_addr learned my addr 172.21.15.38:0/359796640 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:34:39.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdb9d9b700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdbc101f10 0x7fcdbc198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:39.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdba59c700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdbc100d10 0x7fcdbc198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:39.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdbad9d700 1 -- 172.21.15.38:0/359796640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdbc101f10 msgr2=0x7fcdbc198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:39.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdbad9d700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdbc101f10 0x7fcdbc198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdbad9d700 1 -- 172.21.15.38:0/359796640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdbc100d10 msgr2=0x7fcdbc198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:39.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdbad9d700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdbc100d10 0x7fcdbc198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdbad9d700 1 -- 172.21.15.38:0/359796640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc19fda0 con 0x7fcdbc1028b0 2024-03-26T06:34:39.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdb9d9b700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdbc101f10 0x7fcdbc198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:34:39.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdba59c700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdbc100d10 0x7fcdbc198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:34:39.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.481+0000 7fcdbad9d700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1028b0 0x7fcdbc19f700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4007f10 tx=0x7fcdb400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:39.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.482+0000 7fcdb37fe700 1 -- 172.21.15.38:0/359796640 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4004bb0 con 0x7fcdbc1028b0 2024-03-26T06:34:39.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.482+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/359796640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdbc1a0090 con 0x7fcdbc1028b0 2024-03-26T06:34:39.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.482+0000 7fcdb37fe700 1 -- 172.21.15.38:0/359796640 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcdb4004d50 con 0x7fcdbc1028b0 2024-03-26T06:34:39.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.482+0000 7fcdb37fe700 1 -- 172.21.15.38:0/359796640 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdb4005a60 con 0x7fcdbc1028b0 2024-03-26T06:34:39.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.482+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/359796640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdbc1a0580 con 0x7fcdbc1028b0 2024-03-26T06:34:39.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.483+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/359796640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdbc068aa0 con 0x7fcdbc1028b0 2024-03-26T06:34:39.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.484+0000 7fcdb37fe700 1 -- 172.21.15.38:0/359796640 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcdb401a020 con 0x7fcdbc1028b0 2024-03-26T06:34:39.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.485+0000 7fcdb37fe700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcda80a2890 0x7fcda80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:39.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.485+0000 7fcdb37fe700 1 -- 172.21.15.38:0/359796640 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcdb40ce4a0 con 0x7fcdbc1028b0 2024-03-26T06:34:39.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.485+0000 7fcdba59c700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcda80a2890 0x7fcda80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:39.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.485+0000 7fcdba59c700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcda80a2890 0x7fcda80a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcda4000c00 tx=0x7fcda4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:39.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.489+0000 7fcdb37fe700 1 -- 172.21.15.38:0/359796640 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcdb4010070 con 0x7fcdbc1028b0 2024-03-26T06:34:39.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.695+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/359796640 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fcdbc1a0a40 con 0x7fcdbc1028b0 2024-03-26T06:34:39.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.696+0000 7fcdb37fe700 1 -- 172.21.15.38:0/359796640 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) v1 ==== 74+0+133 (secure 0 0 0) 0x7fcdbc1a0a40 con 0x7fcdbc1028b0 2024-03-26T06:34:39.698 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:34:39.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/359796640 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcda80a2890 msgr2=0x7fcda80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:39.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcda80a2890 0x7fcda80a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fcda4000c00 tx=0x7fcda4005fd0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/359796640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1028b0 msgr2=0x7fcdbc19f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:39.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1028b0 0x7fcdbc19f700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4007f10 tx=0x7fcdb400d3b0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/359796640 shutdown_connections 2024-03-26T06:34:39.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdbc1028b0 0x7fcdbc19f700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcda80a2890 0x7fcda80a4d50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcdbc100d10 0x7fcdbc198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 --2- 172.21.15.38:0/359796640 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcdbc101f10 0x7fcdbc198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:39.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/359796640 >> 172.21.15.38:0/359796640 conn(0x7fcdbc0fc2b0 msgr2=0x7fcdbc0fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:39.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/359796640 shutdown_connections 2024-03-26T06:34:39.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:39.698+0000 7fcdc0ad7700 1 -- 172.21.15.38:0/359796640 wait complete. 2024-03-26T06:34:40.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:40 smithi038 ceph-mon[32265]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:40.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:40 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/359796640' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T06:34:40.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:40 smithi121 ceph-mon[36349]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:40.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:40 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/359796640' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T06:34:40.405 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1711434876,"num_in_osds":12,"osd_in_since":1711434856,"num_remapped_pgs":0} 2024-03-26T06:34:40.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T06:34:40.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:40 smithi170 ceph-mon[36912]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:40.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:40 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/359796640' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T06:34:40.649 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:34:42.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 -- 172.21.15.38:0/280818352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc3c101330 msgr2=0x7efc3c106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:42.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 --2- 172.21.15.38:0/280818352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc3c101330 0x7efc3c106a70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0099c0 tx=0x7efc2c009cd0 comp rx=0 tx=0).stop 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 -- 172.21.15.38:0/280818352 shutdown_connections 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 --2- 172.21.15.38:0/280818352 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc3c1071a0 0x7efc3c10ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 --2- 172.21.15.38:0/280818352 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc3c100980 0x7efc3c100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 --2- 172.21.15.38:0/280818352 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc3c101330 0x7efc3c106a70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 -- 172.21.15.38:0/280818352 >> 172.21.15.38:0/280818352 conn(0x7efc3c076020 msgr2=0x7efc3c076430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 -- 172.21.15.38:0/280818352 shutdown_connections 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 -- 172.21.15.38:0/280818352 wait complete. 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 Processor -- start 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 -- start start 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc3c100980 0x7efc3c101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc3c101330 0x7efc3c101e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc3c1071a0 0x7efc3c102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c1120f0 con 0x7efc3c1071a0 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.258+0000 7efc43410700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c111f70 con 0x7efc3c100980 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc43410700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c112270 con 0x7efc3c101330 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc419ad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc3c1071a0 0x7efc3c102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc419ad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc3c1071a0 0x7efc3c102380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43116/0 (socket says 172.21.15.38:43116) 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc419ad700 1 -- 172.21.15.38:0/2307304294 learned_addr learned my addr 172.21.15.38:0/2307304294 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc411ac700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc3c100980 0x7efc3c101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc409ab700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc3c101330 0x7efc3c101e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc419ad700 1 -- 172.21.15.38:0/2307304294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc3c101330 msgr2=0x7efc3c101e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc419ad700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc3c101330 0x7efc3c101e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc419ad700 1 -- 172.21.15.38:0/2307304294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc3c100980 msgr2=0x7efc3c101900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc419ad700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc3c100980 0x7efc3c101900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:42.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc419ad700 1 -- 172.21.15.38:0/2307304294 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c1a6a50 con 0x7efc3c1071a0 2024-03-26T06:34:42.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc419ad700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc3c1071a0 0x7efc3c102380 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efc34007d70 tx=0x7efc3400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:42.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc327fc700 1 -- 172.21.15.38:0/2307304294 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34016020 con 0x7efc3c1071a0 2024-03-26T06:34:42.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc327fc700 1 -- 172.21.15.38:0/2307304294 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc3400ac90 con 0x7efc3c1071a0 2024-03-26T06:34:42.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc327fc700 1 -- 172.21.15.38:0/2307304294 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc340115c0 con 0x7efc3c1071a0 2024-03-26T06:34:42.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.259+0000 7efc43410700 1 -- 172.21.15.38:0/2307304294 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3c1a6d40 con 0x7efc3c1071a0 2024-03-26T06:34:42.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.260+0000 7efc43410700 1 -- 172.21.15.38:0/2307304294 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3c1a7200 con 0x7efc3c1071a0 2024-03-26T06:34:42.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.260+0000 7efc43410700 1 -- 172.21.15.38:0/2307304294 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3c10a3e0 con 0x7efc3c1071a0 2024-03-26T06:34:42.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.261+0000 7efc327fc700 1 -- 172.21.15.38:0/2307304294 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efc34019020 con 0x7efc3c1071a0 2024-03-26T06:34:42.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.262+0000 7efc327fc700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efc280a2890 0x7efc280a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:42.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.262+0000 7efc411ac700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efc280a2890 0x7efc280a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:42.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.262+0000 7efc327fc700 1 -- 172.21.15.38:0/2307304294 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efc340ce540 con 0x7efc3c1071a0 2024-03-26T06:34:42.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.262+0000 7efc411ac700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efc280a2890 0x7efc280a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc38005950 tx=0x7efc3800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:42.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.267+0000 7efc327fc700 1 -- 172.21.15.38:0/2307304294 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7efc34020020 con 0x7efc3c1071a0 2024-03-26T06:34:42.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:42 smithi038 ceph-mon[32265]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:42.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:42 smithi121 ceph-mon[36349]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:42.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:42 smithi170 ceph-mon[36912]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:42.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.472+0000 7efc43410700 1 -- 172.21.15.38:0/2307304294 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc3c109dc0 con 0x7efc3c1071a0 2024-03-26T06:34:42.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.473+0000 7efc327fc700 1 -- 172.21.15.38:0/2307304294 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18446 (secure 0 0 0) 0x7efc3409b9e0 con 0x7efc3c1071a0 2024-03-26T06:34:42.476 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:34:42.476 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":82,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:34:37.602828+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:34:42.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.475+0000 7efc43410700 1 -- 172.21.15.38:0/2307304294 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efc280a2890 msgr2=0x7efc280a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:42.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.475+0000 7efc43410700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efc280a2890 0x7efc280a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7efc38005950 tx=0x7efc3800b410 comp rx=0 tx=0).stop 2024-03-26T06:34:42.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.475+0000 7efc43410700 1 -- 172.21.15.38:0/2307304294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc3c1071a0 msgr2=0x7efc3c102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:42.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.475+0000 7efc43410700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc3c1071a0 0x7efc3c102380 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efc34007d70 tx=0x7efc3400a340 comp rx=0 tx=0).stop 2024-03-26T06:34:42.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.476+0000 7efc43410700 1 -- 172.21.15.38:0/2307304294 shutdown_connections 2024-03-26T06:34:42.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.476+0000 7efc43410700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efc3c1071a0 0x7efc3c102380 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:42.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.476+0000 7efc43410700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efc280a2890 0x7efc280a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:42.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.476+0000 7efc43410700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc3c100980 0x7efc3c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:42.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.476+0000 7efc43410700 1 --2- 172.21.15.38:0/2307304294 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efc3c101330 0x7efc3c101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:42.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.476+0000 7efc43410700 1 -- 172.21.15.38:0/2307304294 >> 172.21.15.38:0/2307304294 conn(0x7efc3c076020 msgr2=0x7efc3c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:42.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.476+0000 7efc43410700 1 -- 172.21.15.38:0/2307304294 shutdown_connections 2024-03-26T06:34:42.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:42.476+0000 7efc43410700 1 -- 172.21.15.38:0/2307304294 wait complete. 2024-03-26T06:34:43.106 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T06:30:52.485897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T06:34:43.106 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T06:34:43.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:43 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2307304294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:34:43.348 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:34:43.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:43 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2307304294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:34:43.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:43 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2307304294' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:34:44.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:44 smithi038 ceph-mon[32265]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:44.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:44 smithi121 ceph-mon[36349]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:44.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:44 smithi170 ceph-mon[36912]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:44.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.877+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2570845368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8c4100ce0 msgr2=0x7ff8c4101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:44.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.877+0000 7ff8c9f3b700 1 --2- 172.21.15.38:0/2570845368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8c4100ce0 0x7ff8c4101140 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8009990 tx=0x7ff8b8009ca0 comp rx=0 tx=0).stop 2024-03-26T06:34:44.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.877+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2570845368 shutdown_connections 2024-03-26T06:34:44.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.877+0000 7ff8c9f3b700 1 --2- 172.21.15.38:0/2570845368 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8c4101680 0x7ff8c410db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:44.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.877+0000 7ff8c9f3b700 1 --2- 172.21.15.38:0/2570845368 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8c4106d20 0x7ff8c4107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:44.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.877+0000 7ff8c9f3b700 1 --2- 172.21.15.38:0/2570845368 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8c4100ce0 0x7ff8c4101140 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:44.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.877+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2570845368 >> 172.21.15.38:0/2570845368 conn(0x7ff8c4077c10 msgr2=0x7ff8c4078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:44.881 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.877+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2570845368 shutdown_connections 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.877+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2570845368 wait complete. 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.878+0000 7ff8c9f3b700 1 Processor -- start 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.878+0000 7ff8c9f3b700 1 -- start start 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.878+0000 7ff8c9f3b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8c4100ce0 0x7ff8c4102990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.878+0000 7ff8c9f3b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8c4101680 0x7ff8c4102ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.878+0000 7ff8c9f3b700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8c4106d20 0x7ff8c4103410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.878+0000 7ff8c9f3b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c4110fe0 con 0x7ff8c4100ce0 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.878+0000 7ff8c9f3b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c4110e60 con 0x7ff8c4101680 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.878+0000 7ff8c9f3b700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c4111160 con 0x7ff8c4106d20 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.878+0000 7ff8c37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8c4100ce0 0x7ff8c4102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.878+0000 7ff8c37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8c4100ce0 0x7ff8c4102990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58324/0 (socket says 172.21.15.38:58324) 2024-03-26T06:34:44.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.878+0000 7ff8c37fe700 1 -- 172.21.15.38:0/2196390988 learned_addr learned my addr 172.21.15.38:0/2196390988 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:34:44.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c3fff700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8c4106d20 0x7ff8c4103410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:44.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c2ffd700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8c4101680 0x7ff8c4102ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:44.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c37fe700 1 -- 172.21.15.38:0/2196390988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8c4106d20 msgr2=0x7ff8c4103410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:44.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c37fe700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8c4106d20 0x7ff8c4103410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:44.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c37fe700 1 -- 172.21.15.38:0/2196390988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8c4101680 msgr2=0x7ff8c4102ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:44.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c37fe700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8c4101680 0x7ff8c4102ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:44.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c37fe700 1 -- 172.21.15.38:0/2196390988 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c41a6b20 con 0x7ff8c4100ce0 2024-03-26T06:34:44.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c37fe700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8c4100ce0 0x7ff8c4102990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400b530 tx=0x7ff8b400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:44.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c0ff9700 1 -- 172.21.15.38:0/2196390988 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4015020 con 0x7ff8c4100ce0 2024-03-26T06:34:44.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c0ff9700 1 -- 172.21.15.38:0/2196390988 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8b4011d10 con 0x7ff8c4100ce0 2024-03-26T06:34:44.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2196390988 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c41a6e10 con 0x7ff8c4100ce0 2024-03-26T06:34:44.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c0ff9700 1 -- 172.21.15.38:0/2196390988 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4010e50 con 0x7ff8c4100ce0 2024-03-26T06:34:44.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.879+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2196390988 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c41a72c0 con 0x7ff8c4100ce0 2024-03-26T06:34:44.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.880+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2196390988 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c4068aa0 con 0x7ff8c4100ce0 2024-03-26T06:34:44.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.881+0000 7ff8c0ff9700 1 -- 172.21.15.38:0/2196390988 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff8b4018020 con 0x7ff8c4100ce0 2024-03-26T06:34:44.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.882+0000 7ff8c0ff9700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8ac0a2a30 0x7ff8ac0a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:44.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.882+0000 7ff8c2ffd700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8ac0a2a30 0x7ff8ac0a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:44.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.882+0000 7ff8c0ff9700 1 -- 172.21.15.38:0/2196390988 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff8b40ce430 con 0x7ff8c4100ce0 2024-03-26T06:34:44.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.883+0000 7ff8c2ffd700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8ac0a2a30 0x7ff8ac0a4ef0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8006010 tx=0x7ff8b800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:44.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:44.886+0000 7ff8c0ff9700 1 -- 172.21.15.38:0/2196390988 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff8b401f020 con 0x7ff8c4100ce0 2024-03-26T06:34:45.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.091+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2196390988 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff8c41a7650 con 0x7ff8c4100ce0 2024-03-26T06:34:45.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.091+0000 7ff8c0ff9700 1 -- 172.21.15.38:0/2196390988 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff8b409b8d0 con 0x7ff8c4100ce0 2024-03-26T06:34:45.093 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-03-26T06:34:45.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.093+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2196390988 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8ac0a2a30 msgr2=0x7ff8ac0a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:45.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.093+0000 7ff8c9f3b700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8ac0a2a30 0x7ff8ac0a4ef0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8006010 tx=0x7ff8b800b540 comp rx=0 tx=0).stop 2024-03-26T06:34:45.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.093+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2196390988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8c4100ce0 msgr2=0x7ff8c4102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:45.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.093+0000 7ff8c9f3b700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8c4100ce0 0x7ff8c4102990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400b530 tx=0x7ff8b400b8f0 comp rx=0 tx=0).stop 2024-03-26T06:34:45.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.094+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2196390988 shutdown_connections 2024-03-26T06:34:45.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.094+0000 7ff8c9f3b700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8c4106d20 0x7ff8c4103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:45.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.094+0000 7ff8c9f3b700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8c4100ce0 0x7ff8c4102990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:45.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.094+0000 7ff8c9f3b700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8ac0a2a30 0x7ff8ac0a4ef0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:45.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.094+0000 7ff8c9f3b700 1 --2- 172.21.15.38:0/2196390988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8c4101680 0x7ff8c4102ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:45.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.094+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2196390988 >> 172.21.15.38:0/2196390988 conn(0x7ff8c4077c10 msgr2=0x7ff8c4100010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:45.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.094+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2196390988 shutdown_connections 2024-03-26T06:34:45.097 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:45.094+0000 7ff8c9f3b700 1 -- 172.21.15.38:0/2196390988 wait complete. 2024-03-26T06:34:45.696 INFO:tasks.cephadm:Adding mds.a on smithi038 2024-03-26T06:34:45.696 INFO:tasks.cephadm:Adding mds.d on smithi038 2024-03-26T06:34:45.696 INFO:tasks.cephadm:Adding mds.g on smithi038 2024-03-26T06:34:45.697 INFO:tasks.cephadm:Adding mds.j on smithi038 2024-03-26T06:34:45.697 INFO:tasks.cephadm:Adding mds.b on smithi121 2024-03-26T06:34:45.697 INFO:tasks.cephadm:Adding mds.e on smithi121 2024-03-26T06:34:45.697 INFO:tasks.cephadm:Adding mds.h on smithi121 2024-03-26T06:34:45.697 INFO:tasks.cephadm:Adding mds.k on smithi121 2024-03-26T06:34:45.697 INFO:tasks.cephadm:Adding mds.c on smithi170 2024-03-26T06:34:45.697 INFO:tasks.cephadm:Adding mds.f on smithi170 2024-03-26T06:34:45.697 INFO:tasks.cephadm:Adding mds.i on smithi170 2024-03-26T06:34:45.697 INFO:tasks.cephadm:Adding mds.l on smithi170 2024-03-26T06:34:45.697 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph orch apply mds all '12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l' 2024-03-26T06:34:45.949 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:34:46.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:46 smithi038 ceph-mon[32265]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:46.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:46 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2196390988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:34:46.374 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:46 smithi170 ceph-mon[36912]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:46.374 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:46 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2196390988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:34:46.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:46 smithi121 ceph-mon[36349]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:46.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:46 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2196390988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:34:47.431 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.429+0000 7fe70207c700 1 -- 172.21.15.170:0/560007766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6fc109ce0 msgr2=0x7fe6fc10c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:47.431 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.429+0000 7fe70207c700 1 --2- 172.21.15.170:0/560007766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6fc109ce0 0x7fe6fc10c170 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe6f400b0a0 tx=0x7fe6f400b3b0 comp rx=0 tx=0).stop 2024-03-26T06:34:47.431 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.429+0000 7fe70207c700 1 -- 172.21.15.170:0/560007766 shutdown_connections 2024-03-26T06:34:47.431 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.429+0000 7fe70207c700 1 --2- 172.21.15.170:0/560007766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6fc109ce0 0x7fe6fc10c170 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:47.431 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.429+0000 7fe70207c700 1 --2- 172.21.15.170:0/560007766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6fc075e10 0x7fe6fc074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:47.431 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.429+0000 7fe70207c700 1 --2- 172.21.15.170:0/560007766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6fc074840 0x7fe6fc074cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:47.431 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.429+0000 7fe70207c700 1 -- 172.21.15.170:0/560007766 >> 172.21.15.170:0/560007766 conn(0x7fe6fc0fd3f0 msgr2=0x7fe6fc0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:47.431 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.429+0000 7fe70207c700 1 -- 172.21.15.170:0/560007766 shutdown_connections 2024-03-26T06:34:47.432 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.429+0000 7fe70207c700 1 -- 172.21.15.170:0/560007766 wait complete. 2024-03-26T06:34:47.434 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.429+0000 7fe70207c700 1 Processor -- start 2024-03-26T06:34:47.434 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.429+0000 7fe70207c700 1 -- start start 2024-03-26T06:34:47.434 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe70207c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6fc074840 0x7fe6fc19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:47.434 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe70207c700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6fc075e10 0x7fe6fc19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:47.434 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe70207c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6fc109ce0 0x7fe6fc1a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:47.434 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe70207c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc10fd10 con 0x7fe6fc074840 2024-03-26T06:34:47.434 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe70207c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc10fb90 con 0x7fe6fc109ce0 2024-03-26T06:34:47.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe70207c700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6fc10fe90 con 0x7fe6fc075e10 2024-03-26T06:34:47.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6faffd700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6fc075e10 0x7fe6fc19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:47.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6faffd700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6fc075e10 0x7fe6fc19a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.170:40684/0 (socket says 172.21.15.170:40684) 2024-03-26T06:34:47.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6faffd700 1 -- 172.21.15.170:0/1989684061 learned_addr learned my addr 172.21.15.170:0/1989684061 (peer_addr_for_me v2:172.21.15.170:0/0) 2024-03-26T06:34:47.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6fb7fe700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6fc074840 0x7fe6fc19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:47.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6fbfff700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6fc109ce0 0x7fe6fc1a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:47.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6faffd700 1 -- 172.21.15.170:0/1989684061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6fc109ce0 msgr2=0x7fe6fc1a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:47.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6faffd700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6fc109ce0 0x7fe6fc1a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:47.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6faffd700 1 -- 172.21.15.170:0/1989684061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6fc074840 msgr2=0x7fe6fc19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:47.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6faffd700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6fc074840 0x7fe6fc19a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:47.435 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6faffd700 1 -- 172.21.15.170:0/1989684061 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc1a1f50 con 0x7fe6fc075e10 2024-03-26T06:34:47.436 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6faffd700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6fc075e10 0x7fe6fc19a830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000b530 tx=0x7fe6f000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:47.436 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6f8ff9700 1 -- 172.21.15.170:0/1989684061 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f0015020 con 0x7fe6fc075e10 2024-03-26T06:34:47.436 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe70207c700 1 -- 172.21.15.170:0/1989684061 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6fc1a2240 con 0x7fe6fc075e10 2024-03-26T06:34:47.436 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6f8ff9700 1 -- 172.21.15.170:0/1989684061 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6f0011d10 con 0x7fe6fc075e10 2024-03-26T06:34:47.436 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.430+0000 7fe6f8ff9700 1 -- 172.21.15.170:0/1989684061 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f000f2e0 con 0x7fe6fc075e10 2024-03-26T06:34:47.436 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.431+0000 7fe70207c700 1 -- 172.21.15.170:0/1989684061 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6fc1a2850 con 0x7fe6fc075e10 2024-03-26T06:34:47.436 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.431+0000 7fe70207c700 1 -- 172.21.15.170:0/1989684061 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6fc068aa0 con 0x7fe6fc075e10 2024-03-26T06:34:47.437 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.432+0000 7fe6f8ff9700 1 -- 172.21.15.170:0/1989684061 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe6f0018020 con 0x7fe6fc075e10 2024-03-26T06:34:47.437 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.433+0000 7fe6f8ff9700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe6e40a2890 0x7fe6e40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:47.437 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.433+0000 7fe6f8ff9700 1 -- 172.21.15.170:0/1989684061 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe6f00cdb70 con 0x7fe6fc075e10 2024-03-26T06:34:47.437 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.433+0000 7fe6fb7fe700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe6e40a2890 0x7fe6e40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:47.437 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.434+0000 7fe6fb7fe700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe6e40a2890 0x7fe6e40a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc0761f0 tx=0x7fe6ec009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:47.439 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.438+0000 7fe6f8ff9700 1 -- 172.21.15.170:0/1989684061 <== mon.1 v2:172.21.15.170:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe6f001f030 con 0x7fe6fc075e10 2024-03-26T06:34:47.643 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.641+0000 7fe70207c700 1 -- 172.21.15.170:0/1989684061 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6fc062320 con 0x7fe6e40a2890 2024-03-26T06:34:47.655 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.653+0000 7fe6f8ff9700 1 -- 172.21.15.170:0/1989684061 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fe6fc062320 con 0x7fe6e40a2890 2024-03-26T06:34:47.656 INFO:teuthology.orchestra.run.smithi170.stdout:Scheduled mds.all update... 2024-03-26T06:34:47.658 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.655+0000 7fe70207c700 1 -- 172.21.15.170:0/1989684061 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe6e40a2890 msgr2=0x7fe6e40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:47.658 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.655+0000 7fe70207c700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe6e40a2890 0x7fe6e40a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc0761f0 tx=0x7fe6ec009190 comp rx=0 tx=0).stop 2024-03-26T06:34:47.658 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.655+0000 7fe70207c700 1 -- 172.21.15.170:0/1989684061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6fc075e10 msgr2=0x7fe6fc19a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:47.658 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.655+0000 7fe70207c700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6fc075e10 0x7fe6fc19a830 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe6f000b530 tx=0x7fe6f000b840 comp rx=0 tx=0).stop 2024-03-26T06:34:47.658 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.655+0000 7fe70207c700 1 -- 172.21.15.170:0/1989684061 shutdown_connections 2024-03-26T06:34:47.658 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.655+0000 7fe70207c700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe6e40a2890 0x7fe6e40a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:47.658 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.655+0000 7fe70207c700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6fc109ce0 0x7fe6fc1a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:47.658 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.655+0000 7fe70207c700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe6fc074840 0x7fe6fc19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:47.659 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.655+0000 7fe70207c700 1 --2- 172.21.15.170:0/1989684061 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe6fc075e10 0x7fe6fc19a830 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:47.659 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.656+0000 7fe70207c700 1 -- 172.21.15.170:0/1989684061 >> 172.21.15.170:0/1989684061 conn(0x7fe6fc0fd3f0 msgr2=0x7fe6fc105be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:47.659 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.656+0000 7fe70207c700 1 -- 172.21.15.170:0/1989684061 shutdown_connections 2024-03-26T06:34:47.659 INFO:teuthology.orchestra.run.smithi170.stderr:2024-03-26T06:34:47.656+0000 7fe70207c700 1 -- 172.21.15.170:0/1989684061 wait complete. 2024-03-26T06:34:48.261 DEBUG:teuthology.orchestra.run.smithi038:mds.a> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.a.service 2024-03-26T06:34:48.264 DEBUG:teuthology.orchestra.run.smithi038:mds.d> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.d.service 2024-03-26T06:34:48.309 DEBUG:teuthology.orchestra.run.smithi038:mds.g> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.g.service 2024-03-26T06:34:48.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:48 smithi038 ceph-mon[32265]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:48.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:48 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:48.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:48 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:48.334 DEBUG:teuthology.orchestra.run.smithi038:mds.j> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.j.service 2024-03-26T06:34:48.377 DEBUG:teuthology.orchestra.run.smithi121:mds.b> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.b.service 2024-03-26T06:34:48.379 DEBUG:teuthology.orchestra.run.smithi121:mds.e> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.e.service 2024-03-26T06:34:48.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:48 smithi121 ceph-mon[36349]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:48.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:48 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:48.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:48 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:48.395 DEBUG:teuthology.orchestra.run.smithi121:mds.h> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.h.service 2024-03-26T06:34:48.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:48 smithi170 ceph-mon[36912]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:48.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:48 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:48.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:48 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:34:48.438 DEBUG:teuthology.orchestra.run.smithi121:mds.k> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.k.service 2024-03-26T06:34:48.481 DEBUG:teuthology.orchestra.run.smithi170:mds.c> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.c.service 2024-03-26T06:34:48.483 DEBUG:teuthology.orchestra.run.smithi170:mds.f> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.f.service 2024-03-26T06:34:48.527 DEBUG:teuthology.orchestra.run.smithi170:mds.i> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.i.service 2024-03-26T06:34:48.570 DEBUG:teuthology.orchestra.run.smithi170:mds.l> sudo journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.l.service 2024-03-26T06:34:48.613 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-26T06:34:48.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:34:48.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.734+0000 7f88a575b640 1 -- 172.21.15.38:0/218469877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88a0100290 msgr2=0x7f88a01006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:48.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.734+0000 7f88a575b640 1 --2- 172.21.15.38:0/218469877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88a0100290 0x7f88a01006b0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8890009900 tx=0x7f889002bc30 comp rx=0 tx=0).stop 2024-03-26T06:34:48.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.734+0000 7f88a575b640 1 -- 172.21.15.38:0/218469877 shutdown_connections 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.734+0000 7f88a575b640 1 --2- 172.21.15.38:0/218469877 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88a0101e40 0x7f88a01086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.734+0000 7f88a575b640 1 --2- 172.21.15.38:0/218469877 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88a0101480 0x7f88a0101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.734+0000 7f88a575b640 1 --2- 172.21.15.38:0/218469877 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88a0100290 0x7f88a01006b0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.734+0000 7f88a575b640 1 -- 172.21.15.38:0/218469877 >> 172.21.15.38:0/218469877 conn(0x7f88a00fb820 msgr2=0x7f88a00fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.734+0000 7f88a575b640 1 -- 172.21.15.38:0/218469877 shutdown_connections 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.734+0000 7f88a575b640 1 -- 172.21.15.38:0/218469877 wait complete. 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.734+0000 7f88a575b640 1 Processor -- start 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f88a575b640 1 -- start start 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f88a575b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88a0100290 0x7f88a006a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f88a575b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88a0101480 0x7f88a019c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f88a575b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88a0101e40 0x7f88a019e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f88a575b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a010beb0 con 0x7f88a0100290 2024-03-26T06:34:48.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f88a575b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a010bd30 con 0x7f88a0101e40 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f88a575b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88a010c030 con 0x7f88a0101480 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f889effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88a0100290 0x7f88a006a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f889effd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88a0100290 0x7f88a006a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58338/0 (socket says 172.21.15.38:58338) 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f889effd640 1 -- 172.21.15.38:0/2653480824 learned_addr learned my addr 172.21.15.38:0/2653480824 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f889f7fe640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88a0101e40 0x7f88a019e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f889e7fc640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88a0101480 0x7f88a019c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f889effd640 1 -- 172.21.15.38:0/2653480824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88a0101480 msgr2=0x7f88a019c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f889effd640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88a0101480 0x7f88a019c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f889effd640 1 -- 172.21.15.38:0/2653480824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88a0101e40 msgr2=0x7f88a019e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f889effd640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88a0101e40 0x7f88a019e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f889effd640 1 -- 172.21.15.38:0/2653480824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a019f2c0 con 0x7f88a0100290 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f889effd640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88a0100290 0x7f88a006a490 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8890000c00 tx=0x7f8890005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:48.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.735+0000 7f8887fff640 1 -- 172.21.15.38:0/2653480824 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890041070 con 0x7f88a0100290 2024-03-26T06:34:48.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.736+0000 7f88a575b640 1 -- 172.21.15.38:0/2653480824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88a019f4f0 con 0x7f88a0100290 2024-03-26T06:34:48.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.736+0000 7f8887fff640 1 -- 172.21.15.38:0/2653480824 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88900044d0 con 0x7f88a0100290 2024-03-26T06:34:48.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.736+0000 7f8887fff640 1 -- 172.21.15.38:0/2653480824 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8890034440 con 0x7f88a0100290 2024-03-26T06:34:48.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.736+0000 7f88a575b640 1 -- 172.21.15.38:0/2653480824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88a019fa30 con 0x7f88a0100290 2024-03-26T06:34:48.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.737+0000 7f88a575b640 1 -- 172.21.15.38:0/2653480824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88a0107630 con 0x7f88a0100290 2024-03-26T06:34:48.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.738+0000 7f8887fff640 1 -- 172.21.15.38:0/2653480824 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8890044070 con 0x7f88a0100290 2024-03-26T06:34:48.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.738+0000 7f8887fff640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88880a2800 0x7f88880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:48.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.738+0000 7f889e7fc640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88880a2800 0x7f88880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:48.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.738+0000 7f8887fff640 1 -- 172.21.15.38:0/2653480824 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f88900f0380 con 0x7f88a0100290 2024-03-26T06:34:48.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.739+0000 7f889e7fc640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88880a2800 0x7f88880a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f88a006a200 tx=0x7f8894005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:48.744 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.742+0000 7f8887fff640 1 -- 172.21.15.38:0/2653480824 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f88900bd8c0 con 0x7f88a0100290 2024-03-26T06:34:48.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.926+0000 7f88a575b640 1 -- 172.21.15.38:0/2653480824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88a002d040 con 0x7f88a0100290 2024-03-26T06:34:48.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.927+0000 7f8887fff640 1 -- 172.21.15.38:0/2653480824 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18446 (secure 0 0 0) 0x7f88900bd1c0 con 0x7f88a0100290 2024-03-26T06:34:48.930 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:34:48.930 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":82,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:34:37.602828+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:34:48.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.929+0000 7f88a575b640 1 -- 172.21.15.38:0/2653480824 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88880a2800 msgr2=0x7f88880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:48.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.929+0000 7f88a575b640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88880a2800 0x7f88880a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f88a006a200 tx=0x7f8894005f90 comp rx=0 tx=0).stop 2024-03-26T06:34:48.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.929+0000 7f88a575b640 1 -- 172.21.15.38:0/2653480824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88a0100290 msgr2=0x7f88a006a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:48.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.929+0000 7f88a575b640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88a0100290 0x7f88a006a490 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8890000c00 tx=0x7f8890005d50 comp rx=0 tx=0).stop 2024-03-26T06:34:48.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.930+0000 7f88a575b640 1 -- 172.21.15.38:0/2653480824 shutdown_connections 2024-03-26T06:34:48.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.930+0000 7f88a575b640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88880a2800 0x7f88880a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:48.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.930+0000 7f88a575b640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88a0101e40 0x7f88a019e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:48.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.930+0000 7f88a575b640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f88a0101480 0x7f88a019c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:48.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.930+0000 7f88a575b640 1 --2- 172.21.15.38:0/2653480824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88a0100290 0x7f88a006a490 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:48.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.930+0000 7f88a575b640 1 -- 172.21.15.38:0/2653480824 >> 172.21.15.38:0/2653480824 conn(0x7f88a00fb820 msgr2=0x7f88a0104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:48.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.930+0000 7f88a575b640 1 -- 172.21.15.38:0/2653480824 shutdown_connections 2024-03-26T06:34:48.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:48.930+0000 7f88a575b640 1 -- 172.21.15.38:0/2653480824 wait complete. 2024-03-26T06:34:48.944 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T06:30:52.485897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T06:34:48.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T06:34:49.051 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 -- 172.21.15.38:0/184466463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5514100210 msgr2=0x7f5514100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 --2- 172.21.15.38:0/184466463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5514100210 0x7f5514100630 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5504009900 tx=0x7f550402bc30 comp rx=0 tx=0).stop 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 -- 172.21.15.38:0/184466463 shutdown_connections 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 --2- 172.21.15.38:0/184466463 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5514101dc0 0x7f5514108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 --2- 172.21.15.38:0/184466463 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5514101400 0x7f5514101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 --2- 172.21.15.38:0/184466463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5514100210 0x7f5514100630 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 -- 172.21.15.38:0/184466463 >> 172.21.15.38:0/184466463 conn(0x7f55140fb7a0 msgr2=0x7f55140fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 -- 172.21.15.38:0/184466463 shutdown_connections 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 -- 172.21.15.38:0/184466463 wait complete. 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 Processor -- start 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 -- start start 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.047+0000 7f551a37e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5514100210 0x7f55141965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:49.052 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551a37e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5514101400 0x7f5514196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:49.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551a37e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5514101dc0 0x7f551419db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:49.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551a37e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551410be70 con 0x7f5514101dc0 2024-03-26T06:34:49.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551a37e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551410bcf0 con 0x7f5514101400 2024-03-26T06:34:49.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551a37e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551410bff0 con 0x7f5514100210 2024-03-26T06:34:49.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551891b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5514101dc0 0x7f551419db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:49.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551891b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5514101dc0 0x7f551419db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58350/0 (socket says 172.21.15.38:58350) 2024-03-26T06:34:49.053 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551891b640 1 -- 172.21.15.38:0/2977322854 learned_addr learned my addr 172.21.15.38:0/2977322854 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:34:49.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f5513fff640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5514100210 0x7f55141965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:49.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f55137fe640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5514101400 0x7f5514196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:49.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551891b640 1 -- 172.21.15.38:0/2977322854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5514100210 msgr2=0x7f55141965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:49.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551891b640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5514100210 0x7f55141965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551891b640 1 -- 172.21.15.38:0/2977322854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5514101400 msgr2=0x7f5514196ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:49.054 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551891b640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5514101400 0x7f5514196ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551891b640 1 -- 172.21.15.38:0/2977322854 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551419e1c0 con 0x7f5514101dc0 2024-03-26T06:34:49.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f55137fe640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5514101400 0x7f5514196ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:34:49.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551891b640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5514101dc0 0x7f551419db50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f550c009fd0 tx=0x7f550c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:49.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f55117fa640 1 -- 172.21.15.38:0/2977322854 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c017070 con 0x7f5514101dc0 2024-03-26T06:34:49.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551a37e640 1 -- 172.21.15.38:0/2977322854 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551419e4b0 con 0x7f5514101dc0 2024-03-26T06:34:49.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f55117fa640 1 -- 172.21.15.38:0/2977322854 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f550c003e50 con 0x7f5514101dc0 2024-03-26T06:34:49.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f55117fa640 1 -- 172.21.15.38:0/2977322854 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c0106c0 con 0x7f5514101dc0 2024-03-26T06:34:49.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.048+0000 7f551a37e640 1 -- 172.21.15.38:0/2977322854 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551419ea00 con 0x7f5514101dc0 2024-03-26T06:34:49.055 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.049+0000 7f551a37e640 1 -- 172.21.15.38:0/2977322854 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55141075a0 con 0x7f5514101dc0 2024-03-26T06:34:49.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.051+0000 7f55117fa640 1 -- 172.21.15.38:0/2977322854 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f550c003970 con 0x7f5514101dc0 2024-03-26T06:34:49.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.051+0000 7f55117fa640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f54fc0a2800 0x7f54fc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:49.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.051+0000 7f5513fff640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f54fc0a2800 0x7f54fc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:49.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.051+0000 7f55117fa640 1 -- 172.21.15.38:0/2977322854 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f550c0ce350 con 0x7f5514101dc0 2024-03-26T06:34:49.056 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.052+0000 7f5513fff640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f54fc0a2800 0x7f54fc0a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5504000c00 tx=0x7f5504005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:49.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.055+0000 7f55117fa640 1 -- 172.21.15.38:0/2977322854 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f550c09b890 con 0x7f5514101dc0 2024-03-26T06:34:49.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.236+0000 7f551a37e640 1 -- 172.21.15.38:0/2977322854 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5514061ef0 con 0x7f5514101dc0 2024-03-26T06:34:49.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.236+0000 7f55117fa640 1 -- 172.21.15.38:0/2977322854 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f550c09b190 con 0x7f5514101dc0 2024-03-26T06:34:49.239 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-03-26T06:34:49.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.238+0000 7f551a37e640 1 -- 172.21.15.38:0/2977322854 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f54fc0a2800 msgr2=0x7f54fc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:49.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.238+0000 7f551a37e640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f54fc0a2800 0x7f54fc0a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5504000c00 tx=0x7f5504005fd0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.238+0000 7f551a37e640 1 -- 172.21.15.38:0/2977322854 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5514101dc0 msgr2=0x7f551419db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:49.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.238+0000 7f551a37e640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5514101dc0 0x7f551419db50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f550c009fd0 tx=0x7f550c00b260 comp rx=0 tx=0).stop 2024-03-26T06:34:49.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.238+0000 7f551a37e640 1 -- 172.21.15.38:0/2977322854 shutdown_connections 2024-03-26T06:34:49.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.238+0000 7f551a37e640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f54fc0a2800 0x7f54fc0a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.238+0000 7f551a37e640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5514101dc0 0x7f551419db50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.239+0000 7f551a37e640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5514101400 0x7f5514196ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.239+0000 7f551a37e640 1 --2- 172.21.15.38:0/2977322854 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5514100210 0x7f55141965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.239+0000 7f551a37e640 1 -- 172.21.15.38:0/2977322854 >> 172.21.15.38:0/2977322854 conn(0x7f55140fb7a0 msgr2=0x7f5514104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:49.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.239+0000 7f551a37e640 1 -- 172.21.15.38:0/2977322854 shutdown_connections 2024-03-26T06:34:49.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.239+0000 7f551a37e640 1 -- 172.21.15.38:0/2977322854 wait complete. 2024-03-26T06:34:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:49 smithi038 ceph-mon[32265]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:34:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:49 smithi038 ceph-mon[32265]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l;count:12 2024-03-26T06:34:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:49 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:49 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2653480824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:34:49.252 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:34:49.254 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:49 smithi170 ceph-mon[36912]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:34:49.254 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:49 smithi170 ceph-mon[36912]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l;count:12 2024-03-26T06:34:49.254 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:49.254 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:49.254 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:49.254 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:49.254 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:49.254 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:49 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:49.254 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:49 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2653480824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:34:49.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.353+0000 7fe163345640 1 -- 172.21.15.38:0/1159906142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe15c1047a0 msgr2=0x7fe15c106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:49.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.353+0000 7fe163345640 1 --2- 172.21.15.38:0/1159906142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe15c1047a0 0x7fe15c106b90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe14c009900 tx=0x7fe14c02bc30 comp rx=0 tx=0).stop 2024-03-26T06:34:49.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.353+0000 7fe163345640 1 -- 172.21.15.38:0/1159906142 shutdown_connections 2024-03-26T06:34:49.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.353+0000 7fe163345640 1 --2- 172.21.15.38:0/1159906142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe15c1070d0 0x7fe15c1094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.353+0000 7fe163345640 1 --2- 172.21.15.38:0/1159906142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe15c1047a0 0x7fe15c106b90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.353+0000 7fe163345640 1 --2- 172.21.15.38:0/1159906142 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c101e70 0x7fe15c104260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.353+0000 7fe163345640 1 -- 172.21.15.38:0/1159906142 >> 172.21.15.38:0/1159906142 conn(0x7fe15c0fb820 msgr2=0x7fe15c0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:49.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.353+0000 7fe163345640 1 -- 172.21.15.38:0/1159906142 shutdown_connections 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.353+0000 7fe163345640 1 -- 172.21.15.38:0/1159906142 wait complete. 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.353+0000 7fe163345640 1 Processor -- start 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.353+0000 7fe163345640 1 -- start start 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe163345640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe15c101e70 0x7fe15c1017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe163345640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c1047a0 0x7fe15c0ffe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe163345640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe15c1070d0 0x7fe15c100360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe163345640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c10cfb0 con 0x7fe15c101e70 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe163345640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c10ce30 con 0x7fe15c1070d0 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe163345640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c10d130 con 0x7fe15c1047a0 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1610e1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe15c101e70 0x7fe15c1017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1610e1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe15c101e70 0x7fe15c1017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58380/0 (socket says 172.21.15.38:58380) 2024-03-26T06:34:49.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1610e1640 1 -- 172.21.15.38:0/2861405040 learned_addr learned my addr 172.21.15.38:0/2861405040 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1608e0640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c1047a0 0x7fe15c0ffe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1610e1640 1 -- 172.21.15.38:0/2861405040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c1047a0 msgr2=0x7fe15c0ffe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1610e1640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c1047a0 0x7fe15c0ffe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1610e1640 1 -- 172.21.15.38:0/2861405040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe15c1070d0 msgr2=0x7fe15c100360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1618e2640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe15c1070d0 0x7fe15c100360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1610e1640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe15c1070d0 0x7fe15c100360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1610e1640 1 -- 172.21.15.38:0/2861405040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c100bf0 con 0x7fe15c101e70 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1610e1640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe15c101e70 0x7fe15c1017e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe15400d770 tx=0x7fe15400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1527fc640 1 -- 172.21.15.38:0/2861405040 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154017070 con 0x7fe15c101e70 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1527fc640 1 -- 172.21.15.38:0/2861405040 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe154005500 con 0x7fe15c101e70 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe163345640 1 -- 172.21.15.38:0/2861405040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe15c1a16f0 con 0x7fe15c101e70 2024-03-26T06:34:49.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe1527fc640 1 -- 172.21.15.38:0/2861405040 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe154004e00 con 0x7fe15c101e70 2024-03-26T06:34:49.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.354+0000 7fe163345640 1 -- 172.21.15.38:0/2861405040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe15c1a1bc0 con 0x7fe15c101e70 2024-03-26T06:34:49.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.355+0000 7fe163345640 1 -- 172.21.15.38:0/2861405040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe15c0fd310 con 0x7fe15c101e70 2024-03-26T06:34:49.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.356+0000 7fe1527fc640 1 -- 172.21.15.38:0/2861405040 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe154005000 con 0x7fe15c101e70 2024-03-26T06:34:49.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.357+0000 7fe1527fc640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1480a2800 0x7fe1480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:49.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.357+0000 7fe1527fc640 1 -- 172.21.15.38:0/2861405040 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe1540ce0b0 con 0x7fe15c101e70 2024-03-26T06:34:49.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.357+0000 7fe1608e0640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1480a2800 0x7fe1480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:49.362 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.357+0000 7fe1608e0640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1480a2800 0x7fe1480a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe14c02c7b0 tx=0x7fe14c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:49.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.361+0000 7fe1527fc640 1 -- 172.21.15.38:0/2861405040 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe15409b5f0 con 0x7fe15c101e70 2024-03-26T06:34:49.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:49 smithi121 ceph-mon[36349]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:34:49.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:49 smithi121 ceph-mon[36349]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l;count:12 2024-03-26T06:34:49.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:49.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:34:49.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:49.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:49.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:49.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:49 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:49.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:49 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2653480824' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:34:49.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.582+0000 7fe163345640 1 -- 172.21.15.38:0/2861405040 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe15c02d100 con 0x7fe15c101e70 2024-03-26T06:34:49.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.583+0000 7fe1527fc640 1 -- 172.21.15.38:0/2861405040 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fe15409aef0 con 0x7fe15c101e70 2024-03-26T06:34:49.585 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:34:49.585 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-26T06:34:49.586 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 1 2024-03-26T06:34:49.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.584+0000 7fe163345640 1 -- 172.21.15.38:0/2861405040 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1480a2800 msgr2=0x7fe1480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:49.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.584+0000 7fe163345640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1480a2800 0x7fe1480a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fe14c02c7b0 tx=0x7fe14c005fd0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.584+0000 7fe163345640 1 -- 172.21.15.38:0/2861405040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe15c101e70 msgr2=0x7fe15c1017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:49.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.584+0000 7fe163345640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe15c101e70 0x7fe15c1017e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe15400d770 tx=0x7fe15400dc30 comp rx=0 tx=0).stop 2024-03-26T06:34:49.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.584+0000 7fe163345640 1 -- 172.21.15.38:0/2861405040 shutdown_connections 2024-03-26T06:34:49.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.584+0000 7fe163345640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1480a2800 0x7fe1480a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.584+0000 7fe163345640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe15c1070d0 0x7fe15c100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.584+0000 7fe163345640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe15c1047a0 0x7fe15c0ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.584+0000 7fe163345640 1 --2- 172.21.15.38:0/2861405040 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe15c101e70 0x7fe15c1017e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:49.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.584+0000 7fe163345640 1 -- 172.21.15.38:0/2861405040 >> 172.21.15.38:0/2861405040 conn(0x7fe15c0fb820 msgr2=0x7fe15c102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:49.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.585+0000 7fe163345640 1 -- 172.21.15.38:0/2861405040 shutdown_connections 2024-03-26T06:34:49.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:49.585+0000 7fe163345640 1 -- 172.21.15.38:0/2861405040 wait complete. 2024-03-26T06:34:50.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:50 smithi038 ceph-mon[32265]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:50.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:50 smithi038 ceph-mon[32265]: Deploying daemon mds.l on smithi170 2024-03-26T06:34:50.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:50 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2977322854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:34:50.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:50 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2861405040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:50.367 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:50 smithi170 ceph-mon[36912]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:50.368 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:50 smithi170 ceph-mon[36912]: Deploying daemon mds.l on smithi170 2024-03-26T06:34:50.368 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:50 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2977322854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:34:50.368 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:50 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2861405040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:50.368 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:34:50 smithi170 systemd[1]: Starting Ceph mds.l for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:34:50.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:50 smithi121 ceph-mon[36349]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:50.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:50 smithi121 ceph-mon[36349]: Deploying daemon mds.l on smithi170 2024-03-26T06:34:50.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:50 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2977322854' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:34:50.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:50 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2861405040' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:50.668 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:34:50 smithi170 podman[67407]: 2024-03-26 06:34:50.483487658 +0000 UTC m=+0.017899757 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:34:50.668 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:34:50 smithi170 podman[67407]: 2024-03-26 06:34:50.583284829 +0000 UTC m=+0.117696927 container create f2e56bf0d4bd703f3fcccde49ca9d0778ff278d523be5ae4c3daaa4d8b6409a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T06:34:51.168 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:34:50 smithi170 podman[67407]: 2024-03-26 06:34:50.793984188 +0000 UTC m=+0.328396289 container init f2e56bf0d4bd703f3fcccde49ca9d0778ff278d523be5ae4c3daaa4d8b6409a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-l, io.buildah.version=1.33.5, ceph=True, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T06:34:51.168 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:34:50 smithi170 podman[67407]: 2024-03-26 06:34:50.796470953 +0000 UTC m=+0.330883046 container start f2e56bf0d4bd703f3fcccde49ca9d0778ff278d523be5ae4c3daaa4d8b6409a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-03-26T06:34:51.168 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:34:50 smithi170 ceph-mds[67426]: -- 172.21.15.170:0/3255556152 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55abfebafa40 con 0x55abfeb82800 2024-03-26T06:34:51.168 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:34:50 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-l[67422]: starting mds.l at 2024-03-26T06:34:51.168 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:34:50 smithi170 bash[67407]: f2e56bf0d4bd703f3fcccde49ca9d0778ff278d523be5ae4c3daaa4d8b6409a8 2024-03-26T06:34:51.168 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:34:50 smithi170 systemd[1]: Started Ceph mds.l for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:34:51.599 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:34:51.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.711+0000 7fd66959f640 1 -- 172.21.15.38:0/2883826280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd664100040 msgr2=0x7fd664100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:51.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.711+0000 7fd66959f640 1 --2- 172.21.15.38:0/2883826280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd664100040 0x7fd664100460 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd654009900 tx=0x7fd65402bc30 comp rx=0 tx=0).stop 2024-03-26T06:34:51.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.711+0000 7fd66959f640 1 -- 172.21.15.38:0/2883826280 shutdown_connections 2024-03-26T06:34:51.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.711+0000 7fd66959f640 1 --2- 172.21.15.38:0/2883826280 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd664101bf0 0x7fd664108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:51.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.711+0000 7fd66959f640 1 --2- 172.21.15.38:0/2883826280 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd664101230 0x7fd6641016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:51.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.711+0000 7fd66959f640 1 --2- 172.21.15.38:0/2883826280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd664100040 0x7fd664100460 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:51.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.711+0000 7fd66959f640 1 -- 172.21.15.38:0/2883826280 >> 172.21.15.38:0/2883826280 conn(0x7fd6640fb5d0 msgr2=0x7fd6640fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:51.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.711+0000 7fd66959f640 1 -- 172.21.15.38:0/2883826280 shutdown_connections 2024-03-26T06:34:51.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.711+0000 7fd66959f640 1 -- 172.21.15.38:0/2883826280 wait complete. 2024-03-26T06:34:51.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.712+0000 7fd66959f640 1 Processor -- start 2024-03-26T06:34:51.715 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.712+0000 7fd66959f640 1 -- start start 2024-03-26T06:34:51.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.712+0000 7fd66959f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd664100040 0x7fd6641974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:51.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.712+0000 7fd66959f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd664101230 0x7fd664197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:51.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.712+0000 7fd66959f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd664101bf0 0x7fd66419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:51.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd66959f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66410bd20 con 0x7fd664100040 2024-03-26T06:34:51.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd66959f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66410bba0 con 0x7fd664101230 2024-03-26T06:34:51.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd66959f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66410bea0 con 0x7fd664101bf0 2024-03-26T06:34:51.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd663fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd664100040 0x7fd6641974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:51.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd663fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd664100040 0x7fd6641974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58398/0 (socket says 172.21.15.38:58398) 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd663fff640 1 -- 172.21.15.38:0/305470028 learned_addr learned my addr 172.21.15.38:0/305470028 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd668d9e640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd664101bf0 0x7fd66419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd6637fe640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd664101230 0x7fd664197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd663fff640 1 -- 172.21.15.38:0/305470028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd664101bf0 msgr2=0x7fd66419ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd663fff640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd664101bf0 0x7fd66419ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd663fff640 1 -- 172.21.15.38:0/305470028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd664101230 msgr2=0x7fd664197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd663fff640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd664101230 0x7fd664197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd663fff640 1 -- 172.21.15.38:0/305470028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66419f0b0 con 0x7fd664100040 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd663fff640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd664100040 0x7fd6641974e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd654000c00 tx=0x7fd65402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd6617fa640 1 -- 172.21.15.38:0/305470028 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd654043020 con 0x7fd664100040 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd66959f640 1 -- 172.21.15.38:0/305470028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66419f340 con 0x7fd664100040 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd6617fa640 1 -- 172.21.15.38:0/305470028 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6540054e0 con 0x7fd664100040 2024-03-26T06:34:51.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd6617fa640 1 -- 172.21.15.38:0/305470028 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd654003ce0 con 0x7fd664100040 2024-03-26T06:34:51.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.713+0000 7fd66959f640 1 -- 172.21.15.38:0/305470028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66419f860 con 0x7fd664100040 2024-03-26T06:34:51.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.715+0000 7fd66959f640 1 -- 172.21.15.38:0/305470028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66419fa60 con 0x7fd664100040 2024-03-26T06:34:51.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.715+0000 7fd6617fa640 1 -- 172.21.15.38:0/305470028 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd654046020 con 0x7fd664100040 2024-03-26T06:34:51.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.716+0000 7fd6617fa640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd64c0a2800 0x7fd64c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:51.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.716+0000 7fd6637fe640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd64c0a2800 0x7fd64c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:51.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.716+0000 7fd6617fa640 1 -- 172.21.15.38:0/305470028 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd6540f06c0 con 0x7fd664100040 2024-03-26T06:34:51.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.717+0000 7fd6637fe640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd64c0a2800 0x7fd64c0a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd6641989a0 tx=0x7fd65800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:51.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.721+0000 7fd6617fa640 1 -- 172.21.15.38:0/305470028 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd65404d020 con 0x7fd664100040 2024-03-26T06:34:51.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.941+0000 7fd66959f640 1 -- 172.21.15.38:0/305470028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd66419fd10 con 0x7fd664100040 2024-03-26T06:34:51.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.942+0000 7fd6617fa640 1 -- 172.21.15.38:0/305470028 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7fd6540bdc00 con 0x7fd664100040 2024-03-26T06:34:51.944 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:34:51.944 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-26T06:34:51.945 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 2 2024-03-26T06:34:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 -- 172.21.15.38:0/305470028 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd64c0a2800 msgr2=0x7fd64c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd64c0a2800 0x7fd64c0a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd6641989a0 tx=0x7fd65800b410 comp rx=0 tx=0).stop 2024-03-26T06:34:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 -- 172.21.15.38:0/305470028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd664100040 msgr2=0x7fd6641974e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd664100040 0x7fd6641974e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd654000c00 tx=0x7fd65402c6c0 comp rx=0 tx=0).stop 2024-03-26T06:34:51.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 -- 172.21.15.38:0/305470028 shutdown_connections 2024-03-26T06:34:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd64c0a2800 0x7fd64c0a4cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd664101bf0 0x7fd66419ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd664101230 0x7fd664197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 --2- 172.21.15.38:0/305470028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd664100040 0x7fd6641974e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 -- 172.21.15.38:0/305470028 >> 172.21.15.38:0/305470028 conn(0x7fd6640fb5d0 msgr2=0x7fd6640fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 -- 172.21.15.38:0/305470028 shutdown_connections 2024-03-26T06:34:51.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:51.944+0000 7fd66959f640 1 -- 172.21.15.38:0/305470028 wait complete. 2024-03-26T06:34:51.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:51 smithi038 ceph-mon[32265]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:51.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:51 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.170:6832/1336487813,v1:172.21.15.170:6833/1336487813] up:boot 2024-03-26T06:34:51.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:51 smithi038 ceph-mon[32265]: fsmap 1 up:standby 2024-03-26T06:34:51.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:51 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-26T06:34:51.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:51 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:51.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:51 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:51.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:51 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:51.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:51 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:51.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:51 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:51.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:51 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:51.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:51 smithi038 ceph-mon[32265]: Deploying daemon mds.k on smithi121 2024-03-26T06:34:52.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:51 smithi121 ceph-mon[36349]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:52.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:51 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.170:6832/1336487813,v1:172.21.15.170:6833/1336487813] up:boot 2024-03-26T06:34:52.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:51 smithi121 ceph-mon[36349]: fsmap 1 up:standby 2024-03-26T06:34:52.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:51 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-26T06:34:52.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:51 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:52.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:51 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:52.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:51 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:52.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:51 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:52.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:51 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:52.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:51 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:52.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:51 smithi121 ceph-mon[36349]: Deploying daemon mds.k on smithi121 2024-03-26T06:34:52.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:51 smithi170 ceph-mon[36912]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:52.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:51 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.170:6832/1336487813,v1:172.21.15.170:6833/1336487813] up:boot 2024-03-26T06:34:52.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:51 smithi170 ceph-mon[36912]: fsmap 1 up:standby 2024-03-26T06:34:52.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:51 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-26T06:34:52.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:51 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:52.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:51 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:52.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:51 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:52.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:51 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:52.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:51 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:52.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:51 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:52.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:51 smithi170 ceph-mon[36912]: Deploying daemon mds.k on smithi121 2024-03-26T06:34:53.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:52 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/305470028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:53.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:52 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/305470028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:53.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:52 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/305470028' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:53.626 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:34:53 smithi121 systemd[1]: Starting Ceph mds.k for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:34:53.877 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:53 smithi121 ceph-mon[36349]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:53.877 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:34:53 smithi121 podman[65844]: 2024-03-26 06:34:53.526037175 +0000 UTC m=+0.017605590 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:34:53.877 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:34:53 smithi121 podman[65844]: 2024-03-26 06:34:53.658431693 +0000 UTC m=+0.150000107 container create 15dddf100d4a0c490d13c1ef32881f1fa1b084769d0b0f8fe9d7f64ed291b142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-k, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T06:34:53.877 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:34:53 smithi121 podman[65844]: 2024-03-26 06:34:53.875732444 +0000 UTC m=+0.367300858 container init 15dddf100d4a0c490d13c1ef32881f1fa1b084769d0b0f8fe9d7f64ed291b142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-k, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319) 2024-03-26T06:34:53.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:34:54.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.077+0000 7fa4fbe43640 1 -- 172.21.15.38:0/688193006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4f4106ff0 msgr2=0x7fa4f4109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:54.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.077+0000 7fa4fbe43640 1 --2- 172.21.15.38:0/688193006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4f4106ff0 0x7fa4f4109440 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec009ca0 tx=0x7fa4ec02c0f0 comp rx=0 tx=0).stop 2024-03-26T06:34:54.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.077+0000 7fa4fbe43640 1 -- 172.21.15.38:0/688193006 shutdown_connections 2024-03-26T06:34:54.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.077+0000 7fa4fbe43640 1 --2- 172.21.15.38:0/688193006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4f4106ff0 0x7fa4f4109440 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:54.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.077+0000 7fa4fbe43640 1 --2- 172.21.15.38:0/688193006 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f41046c0 0x7fa4f4106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:54.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.077+0000 7fa4fbe43640 1 --2- 172.21.15.38:0/688193006 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4f4101d90 0x7fa4f4104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:54.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.077+0000 7fa4fbe43640 1 -- 172.21.15.38:0/688193006 >> 172.21.15.38:0/688193006 conn(0x7fa4f40fb780 msgr2=0x7fa4f40fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:54.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.077+0000 7fa4fbe43640 1 -- 172.21.15.38:0/688193006 shutdown_connections 2024-03-26T06:34:54.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.077+0000 7fa4fbe43640 1 -- 172.21.15.38:0/688193006 wait complete. 2024-03-26T06:34:54.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4fbe43640 1 Processor -- start 2024-03-26T06:34:54.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4fbe43640 1 -- start start 2024-03-26T06:34:54.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4fbe43640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4101d90 0x7fa4f41975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4fbe43640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4f41046c0 0x7fa4f4197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4fbe43640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4f4106ff0 0x7fa4f419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4fbe43640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f410cfa0 con 0x7fa4f41046c0 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4fbe43640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f410ce20 con 0x7fa4f4101d90 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4fbe43640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f410d120 con 0x7fa4f4106ff0 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4f93de640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4f41046c0 0x7fa4f4197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4f93de640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4f41046c0 0x7fa4f4197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58422/0 (socket says 172.21.15.38:58422) 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4f9bdf640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4101d90 0x7fa4f41975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4f93de640 1 -- 172.21.15.38:0/3622852827 learned_addr learned my addr 172.21.15.38:0/3622852827 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4fa3e0640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4f4106ff0 0x7fa4f419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4f9bdf640 1 -- 172.21.15.38:0/3622852827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4f4106ff0 msgr2=0x7fa4f419eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4f9bdf640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4f4106ff0 0x7fa4f419eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:54.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4f9bdf640 1 -- 172.21.15.38:0/3622852827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4f41046c0 msgr2=0x7fa4f4197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4f9bdf640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4f41046c0 0x7fa4f4197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4f9bdf640 1 -- 172.21.15.38:0/3622852827 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f419f210 con 0x7fa4f4101d90 2024-03-26T06:34:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.078+0000 7fa4f93de640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4f41046c0 0x7fa4f4197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:34:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.079+0000 7fa4f9bdf640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4101d90 0x7fa4f41975f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800e9f0 tx=0x7fa4e800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.080+0000 7fa4e6ffd640 1 -- 172.21.15.38:0/3622852827 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e800ccc0 con 0x7fa4f4101d90 2024-03-26T06:34:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.080+0000 7fa4e6ffd640 1 -- 172.21.15.38:0/3622852827 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4e8005020 con 0x7fa4f4101d90 2024-03-26T06:34:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.080+0000 7fa4e6ffd640 1 -- 172.21.15.38:0/3622852827 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e80049e0 con 0x7fa4f4101d90 2024-03-26T06:34:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.080+0000 7fa4fbe43640 1 -- 172.21.15.38:0/3622852827 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f419f500 con 0x7fa4f4101d90 2024-03-26T06:34:54.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.080+0000 7fa4fbe43640 1 -- 172.21.15.38:0/3622852827 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f419fa50 con 0x7fa4f4101d90 2024-03-26T06:34:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.081+0000 7fa4fbe43640 1 -- 172.21.15.38:0/3622852827 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f40fd270 con 0x7fa4f4101d90 2024-03-26T06:34:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.082+0000 7fa4e6ffd640 1 -- 172.21.15.38:0/3622852827 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa4e8004c00 con 0x7fa4f4101d90 2024-03-26T06:34:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.082+0000 7fa4e6ffd640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa4e00a2880 0x7fa4e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.082+0000 7fa4f93de640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa4e00a2880 0x7fa4e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:54.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.082+0000 7fa4e6ffd640 1 -- 172.21.15.38:0/3622852827 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa4e80ce4e0 con 0x7fa4f4101d90 2024-03-26T06:34:54.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.083+0000 7fa4f93de640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa4e00a2880 0x7fa4e00a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4198b00 tx=0x7fa4f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:54.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.087+0000 7fa4e6ffd640 1 -- 172.21.15.38:0/3622852827 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa4e809ba20 con 0x7fa4f4101d90 2024-03-26T06:34:54.145 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:34:53 smithi121 podman[65844]: 2024-03-26 06:34:53.879016194 +0000 UTC m=+0.370584616 container start 15dddf100d4a0c490d13c1ef32881f1fa1b084769d0b0f8fe9d7f64ed291b142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-k, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-26T06:34:54.145 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:34:53 smithi121 ceph-mds[65863]: -- 172.21.15.121:0/2347108439 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559c00d95a40 con 0x559c00d69000 2024-03-26T06:34:54.145 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:34:53 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-k[65859]: starting mds.k at 2024-03-26T06:34:54.145 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:34:54 smithi121 bash[65844]: 15dddf100d4a0c490d13c1ef32881f1fa1b084769d0b0f8fe9d7f64ed291b142 2024-03-26T06:34:54.145 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:34:54 smithi121 systemd[1]: Started Ceph mds.k for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:34:54.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:53 smithi170 ceph-mon[36912]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:54.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.310+0000 7fa4fbe43640 1 -- 172.21.15.38:0/3622852827 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4f4061ef0 con 0x7fa4f4101d90 2024-03-26T06:34:54.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.311+0000 7fa4e6ffd640 1 -- 172.21.15.38:0/3622852827 <== mon.2 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7fa4e80b9e20 con 0x7fa4f4101d90 2024-03-26T06:34:54.314 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:34:54.315 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-26T06:34:54.315 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 2 2024-03-26T06:34:54.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 -- 172.21.15.38:0/3622852827 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa4e00a2880 msgr2=0x7fa4e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:54.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa4e00a2880 0x7fa4e00a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4198b00 tx=0x7fa4f0005f90 comp rx=0 tx=0).stop 2024-03-26T06:34:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 -- 172.21.15.38:0/3622852827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4101d90 msgr2=0x7fa4f41975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4101d90 0x7fa4f41975f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa4e800e9f0 tx=0x7fa4e800eeb0 comp rx=0 tx=0).stop 2024-03-26T06:34:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 -- 172.21.15.38:0/3622852827 shutdown_connections 2024-03-26T06:34:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa4e00a2880 0x7fa4e00a4d40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa4f4106ff0 0x7fa4f419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa4f41046c0 0x7fa4f4197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 --2- 172.21.15.38:0/3622852827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa4f4101d90 0x7fa4f41975f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 -- 172.21.15.38:0/3622852827 >> 172.21.15.38:0/3622852827 conn(0x7fa4f40fb780 msgr2=0x7fa4f41026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 -- 172.21.15.38:0/3622852827 shutdown_connections 2024-03-26T06:34:54.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:54.313+0000 7fa4fbe43640 1 -- 172.21.15.38:0/3622852827 wait complete. 2024-03-26T06:34:54.326 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:53 smithi038 ceph-mon[32265]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:55.319 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:55 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:55.319 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:55 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:55.319 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:55 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:55.319 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:55 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:55.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:55 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:55.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:55 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:55.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:55 smithi121 ceph-mon[36349]: Deploying daemon mds.j on smithi038 2024-03-26T06:34:55.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:55 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3622852827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:55.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:55 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.121:6832/2121798105,v1:172.21.15.121:6833/2121798105] up:boot 2024-03-26T06:34:55.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:55 smithi121 ceph-mon[36349]: fsmap 2 up:standby 2024-03-26T06:34:55.320 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:55 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-26T06:34:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:55.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:55 smithi038 ceph-mon[32265]: Deploying daemon mds.j on smithi038 2024-03-26T06:34:55.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:55 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3622852827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:55.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:55 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.121:6832/2121798105,v1:172.21.15.121:6833/2121798105] up:boot 2024-03-26T06:34:55.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:55 smithi038 ceph-mon[32265]: fsmap 2 up:standby 2024-03-26T06:34:55.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:55 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-26T06:34:55.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:55 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:55.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:55 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:55.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:55 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:55.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:55 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:55.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:55 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:55.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:55 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:55.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:55 smithi170 ceph-mon[36912]: Deploying daemon mds.j on smithi038 2024-03-26T06:34:55.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:55 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3622852827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:55.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:55 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.121:6832/2121798105,v1:172.21.15.121:6833/2121798105] up:boot 2024-03-26T06:34:55.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:55 smithi170 ceph-mon[36912]: fsmap 2 up:standby 2024-03-26T06:34:55.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:55 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-26T06:34:56.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:34:56.357 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:56 smithi038 ceph-mon[32265]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:56.394 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:56 smithi121 ceph-mon[36349]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:56.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:56 smithi170 ceph-mon[36912]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:56.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.529+0000 7fe1bca27640 1 -- 172.21.15.38:0/514515627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1b8101dc0 msgr2=0x7fe1b8108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.529+0000 7fe1bca27640 1 --2- 172.21.15.38:0/514515627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1b8101dc0 0x7fe1b8108650 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8009860 tx=0x7fe1a802bb70 comp rx=0 tx=0).stop 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.529+0000 7fe1bca27640 1 -- 172.21.15.38:0/514515627 shutdown_connections 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.529+0000 7fe1bca27640 1 --2- 172.21.15.38:0/514515627 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1b8101dc0 0x7fe1b8108650 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.529+0000 7fe1bca27640 1 --2- 172.21.15.38:0/514515627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1b8101400 0x7fe1b8101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.529+0000 7fe1bca27640 1 --2- 172.21.15.38:0/514515627 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1b8100210 0x7fe1b8100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.529+0000 7fe1bca27640 1 -- 172.21.15.38:0/514515627 >> 172.21.15.38:0/514515627 conn(0x7fe1b80fb780 msgr2=0x7fe1b80fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.530+0000 7fe1bca27640 1 -- 172.21.15.38:0/514515627 shutdown_connections 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.530+0000 7fe1bca27640 1 -- 172.21.15.38:0/514515627 wait complete. 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.530+0000 7fe1bca27640 1 Processor -- start 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.530+0000 7fe1bca27640 1 -- start start 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.530+0000 7fe1bca27640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1b8100210 0x7fe1b81975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:56.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.530+0000 7fe1bca27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1b8101400 0x7fe1b8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.530+0000 7fe1bca27640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1b8101dc0 0x7fe1b819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.530+0000 7fe1bca27640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b810bea0 con 0x7fe1b8101400 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.530+0000 7fe1bca27640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b810bd20 con 0x7fe1b8100210 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.530+0000 7fe1bca27640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b810c020 con 0x7fe1b8101dc0 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1b5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1b8101400 0x7fe1b8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1b5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1b8101400 0x7fe1b8197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58456/0 (socket says 172.21.15.38:58456) 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1b5d9b640 1 -- 172.21.15.38:0/1296818982 learned_addr learned my addr 172.21.15.38:0/1296818982 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1b659c640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1b8100210 0x7fe1b81975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1b6d9d640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1b8101dc0 0x7fe1b819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1b5d9b640 1 -- 172.21.15.38:0/1296818982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1b8101dc0 msgr2=0x7fe1b819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1b5d9b640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1b8101dc0 0x7fe1b819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1b5d9b640 1 -- 172.21.15.38:0/1296818982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1b8100210 msgr2=0x7fe1b81975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:56.538 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1b5d9b640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1b8100210 0x7fe1b81975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:56.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1b5d9b640 1 -- 172.21.15.38:0/1296818982 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b819f1e0 con 0x7fe1b8101400 2024-03-26T06:34:56.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1b5d9b640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1b8101400 0x7fe1b8197b00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00c420 tx=0x7fe1ac009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:56.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1a77fe640 1 -- 172.21.15.38:0/1296818982 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac003970 con 0x7fe1b8101400 2024-03-26T06:34:56.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1bca27640 1 -- 172.21.15.38:0/1296818982 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b819f4d0 con 0x7fe1b8101400 2024-03-26T06:34:56.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1bca27640 1 -- 172.21.15.38:0/1296818982 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b819fa70 con 0x7fe1b8101400 2024-03-26T06:34:56.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.531+0000 7fe1a77fe640 1 -- 172.21.15.38:0/1296818982 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1ac0097e0 con 0x7fe1b8101400 2024-03-26T06:34:56.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.532+0000 7fe1a77fe640 1 -- 172.21.15.38:0/1296818982 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1ac003970 con 0x7fe1b8101400 2024-03-26T06:34:56.539 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.533+0000 7fe1bca27640 1 -- 172.21.15.38:0/1296818982 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe198005290 con 0x7fe1b8101400 2024-03-26T06:34:56.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.534+0000 7fe1a77fe640 1 -- 172.21.15.38:0/1296818982 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe1ac01d020 con 0x7fe1b8101400 2024-03-26T06:34:56.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.535+0000 7fe1a77fe640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1a00a2a00 0x7fe1a00a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:56.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.535+0000 7fe1a77fe640 1 -- 172.21.15.38:0/1296818982 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe1ac0ce4b0 con 0x7fe1b8101400 2024-03-26T06:34:56.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.535+0000 7fe1b659c640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1a00a2a00 0x7fe1a00a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:56.540 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.535+0000 7fe1b659c640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1a00a2a00 0x7fe1a00a4ec0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8101260 tx=0x7fe1b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:56.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.539+0000 7fe1a77fe640 1 -- 172.21.15.38:0/1296818982 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe1ac09b9f0 con 0x7fe1b8101400 2024-03-26T06:34:56.687 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:34:56 smithi038 systemd[1]: Starting Ceph mds.j for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:34:56.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.768+0000 7fe1bca27640 1 -- 172.21.15.38:0/1296818982 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe198005580 con 0x7fe1b8101400 2024-03-26T06:34:56.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.768+0000 7fe1a77fe640 1 -- 172.21.15.38:0/1296818982 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7fe1ac09b2f0 con 0x7fe1b8101400 2024-03-26T06:34:56.772 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:34:56.772 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-26T06:34:56.772 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 3 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 -- 172.21.15.38:0/1296818982 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1a00a2a00 msgr2=0x7fe1a00a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1a00a2a00 0x7fe1a00a4ec0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe1b8101260 tx=0x7fe1b0005f90 comp rx=0 tx=0).stop 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 -- 172.21.15.38:0/1296818982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1b8101400 msgr2=0x7fe1b8197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1b8101400 0x7fe1b8197b00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe1ac00c420 tx=0x7fe1ac009f90 comp rx=0 tx=0).stop 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 -- 172.21.15.38:0/1296818982 shutdown_connections 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe1a00a2a00 0x7fe1a00a4ec0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe1b8101dc0 0x7fe1b819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1b8101400 0x7fe1b8197b00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 --2- 172.21.15.38:0/1296818982 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1b8100210 0x7fe1b81975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 -- 172.21.15.38:0/1296818982 >> 172.21.15.38:0/1296818982 conn(0x7fe1b80fb780 msgr2=0x7fe1b8068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 -- 172.21.15.38:0/1296818982 shutdown_connections 2024-03-26T06:34:56.774 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:56.770+0000 7fe1bca27640 1 -- 172.21.15.38:0/1296818982 wait complete. 2024-03-26T06:34:56.957 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:34:56 smithi038 podman[70292]: 2024-03-26 06:34:56.590009522 +0000 UTC m=+0.020494766 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:34:56.957 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:34:56 smithi038 podman[70292]: 2024-03-26 06:34:56.71036946 +0000 UTC m=+0.140854695 container create 386c31cfabf7995642a2d4f067f1af14f6856828773921b84a54bf04cc52f7de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-j, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-26T06:34:56.957 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:34:56 smithi038 podman[70292]: 2024-03-26 06:34:56.929803115 +0000 UTC m=+0.360288350 container init 386c31cfabf7995642a2d4f067f1af14f6856828773921b84a54bf04cc52f7de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T06:34:56.957 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:34:56 smithi038 podman[70292]: 2024-03-26 06:34:56.932489455 +0000 UTC m=+0.362974691 container start 386c31cfabf7995642a2d4f067f1af14f6856828773921b84a54bf04cc52f7de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-j, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T06:34:56.957 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:34:56 smithi038 ceph-mds[70313]: -- 172.21.15.38:0/1869027172 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555896b17a40 con 0x555896aea800 2024-03-26T06:34:57.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:57 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1296818982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:57.333 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:34:56 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-j[70309]: starting mds.j at 2024-03-26T06:34:57.333 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:34:57 smithi038 bash[70292]: 386c31cfabf7995642a2d4f067f1af14f6856828773921b84a54bf04cc52f7de 2024-03-26T06:34:57.333 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:34:57 smithi038 systemd[1]: Started Ceph mds.j for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:34:57.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:57 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1296818982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:57.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:57 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1296818982' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:34:58.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:58 smithi038 ceph-mon[32265]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:58.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:58.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:58 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.38:6834/4105682236,v1:172.21.15.38:6835/4105682236] up:boot 2024-03-26T06:34:58.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:58 smithi038 ceph-mon[32265]: fsmap 3 up:standby 2024-03-26T06:34:58.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-26T06:34:58.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:58.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:58.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:58.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:58.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:58 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:58.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:34:58 smithi038 ceph-mon[32265]: Deploying daemon mds.d on smithi038 2024-03-26T06:34:58.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:58 smithi121 ceph-mon[36349]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:58.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:58 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:58.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:58 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.38:6834/4105682236,v1:172.21.15.38:6835/4105682236] up:boot 2024-03-26T06:34:58.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:58 smithi121 ceph-mon[36349]: fsmap 3 up:standby 2024-03-26T06:34:58.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:58 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-26T06:34:58.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:58 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:58.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:58 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:58.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:58 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:58.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:58 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:58.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:58 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:58.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:34:58 smithi121 ceph-mon[36349]: Deploying daemon mds.d on smithi038 2024-03-26T06:34:58.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:58 smithi170 ceph-mon[36912]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:34:58.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:58 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:58.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:58 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.38:6834/4105682236,v1:172.21.15.38:6835/4105682236] up:boot 2024-03-26T06:34:58.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:58 smithi170 ceph-mon[36912]: fsmap 3 up:standby 2024-03-26T06:34:58.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:58 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-26T06:34:58.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:58 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:58.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:58 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:34:58.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:58 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:34:58.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:58 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:34:58.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:58 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:34:58.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:34:58 smithi170 ceph-mon[36912]: Deploying daemon mds.d on smithi038 2024-03-26T06:34:58.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:34:58.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.899+0000 7f62d7fe6640 1 -- 172.21.15.38:0/4199682761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62d01046c0 msgr2=0x7f62d0106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:58.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.899+0000 7f62d7fe6640 1 --2- 172.21.15.38:0/4199682761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62d01046c0 0x7f62d0106ab0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009900 tx=0x7f62c002bc30 comp rx=0 tx=0).stop 2024-03-26T06:34:58.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.899+0000 7f62d7fe6640 1 -- 172.21.15.38:0/4199682761 shutdown_connections 2024-03-26T06:34:58.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.899+0000 7f62d7fe6640 1 --2- 172.21.15.38:0/4199682761 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d0106ff0 0x7f62d0109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:58.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.899+0000 7f62d7fe6640 1 --2- 172.21.15.38:0/4199682761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62d01046c0 0x7f62d0106ab0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:58.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.899+0000 7f62d7fe6640 1 --2- 172.21.15.38:0/4199682761 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d0101d90 0x7f62d0104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:58.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.899+0000 7f62d7fe6640 1 -- 172.21.15.38:0/4199682761 >> 172.21.15.38:0/4199682761 conn(0x7f62d00fb780 msgr2=0x7f62d00fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:58.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.899+0000 7f62d7fe6640 1 -- 172.21.15.38:0/4199682761 shutdown_connections 2024-03-26T06:34:58.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.899+0000 7f62d7fe6640 1 -- 172.21.15.38:0/4199682761 wait complete. 2024-03-26T06:34:58.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d7fe6640 1 Processor -- start 2024-03-26T06:34:58.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d7fe6640 1 -- start start 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d7fe6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62d0101d90 0x7f62d0197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d7fe6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d01046c0 0x7f62d0197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d7fe6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d0106ff0 0x7f62d019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d7fe6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d010cff0 con 0x7f62d0101d90 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d7fe6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d010ce70 con 0x7f62d0106ff0 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d7fe6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d010d170 con 0x7f62d01046c0 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d5d82640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62d0101d90 0x7f62d0197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d5d82640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62d0101d90 0x7f62d0197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58482/0 (socket says 172.21.15.38:58482) 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d5d82640 1 -- 172.21.15.38:0/1557015238 learned_addr learned my addr 172.21.15.38:0/1557015238 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d6583640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d0106ff0 0x7f62d019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d5581640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d01046c0 0x7f62d0197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d5d82640 1 -- 172.21.15.38:0/1557015238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d01046c0 msgr2=0x7f62d0197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d5d82640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d01046c0 0x7f62d0197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:58.905 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d5d82640 1 -- 172.21.15.38:0/1557015238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d0106ff0 msgr2=0x7f62d019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:58.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d5d82640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d0106ff0 0x7f62d019ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:58.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d5d82640 1 -- 172.21.15.38:0/1557015238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d019f260 con 0x7f62d0101d90 2024-03-26T06:34:58.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62d5d82640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62d0101d90 0x7f62d0197640 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00c970 tx=0x7f62cc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:58.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.900+0000 7f62c6ffd640 1 -- 172.21.15.38:0/1557015238 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc007bb0 con 0x7f62d0101d90 2024-03-26T06:34:58.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.901+0000 7f62c6ffd640 1 -- 172.21.15.38:0/1557015238 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62cc007d50 con 0x7f62d0101d90 2024-03-26T06:34:58.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.901+0000 7f62c6ffd640 1 -- 172.21.15.38:0/1557015238 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62cc00f450 con 0x7f62d0101d90 2024-03-26T06:34:58.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.901+0000 7f62d7fe6640 1 -- 172.21.15.38:0/1557015238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d019f550 con 0x7f62d0101d90 2024-03-26T06:34:58.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.901+0000 7f62d7fe6640 1 -- 172.21.15.38:0/1557015238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d019fa20 con 0x7f62d0101d90 2024-03-26T06:34:58.906 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.902+0000 7f62d7fe6640 1 -- 172.21.15.38:0/1557015238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d00fd270 con 0x7f62d0101d90 2024-03-26T06:34:58.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.903+0000 7f62c6ffd640 1 -- 172.21.15.38:0/1557015238 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f62cc004fe0 con 0x7f62d0101d90 2024-03-26T06:34:58.907 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.903+0000 7f62c6ffd640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62bc0a2800 0x7f62bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:34:58.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.903+0000 7f62d5581640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62bc0a2800 0x7f62bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:34:58.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.903+0000 7f62c6ffd640 1 -- 172.21.15.38:0/1557015238 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f62cc0ce7b0 con 0x7f62d0101d90 2024-03-26T06:34:58.908 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.904+0000 7f62d5581640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62bc0a2800 0x7f62bc0a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f62c002c170 tx=0x7f62c002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:34:58.909 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:58.907+0000 7f62c6ffd640 1 -- 172.21.15.38:0/1557015238 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f62cc014030 con 0x7f62d0101d90 2024-03-26T06:34:59.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.140+0000 7f62d7fe6640 1 -- 172.21.15.38:0/1557015238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62d002ce70 con 0x7f62d0101d90 2024-03-26T06:34:59.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.141+0000 7f62c6ffd640 1 -- 172.21.15.38:0/1557015238 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2913 (secure 0 0 0) 0x7f62cc09bcf0 con 0x7f62d0101d90 2024-03-26T06:34:59.143 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:34:59.143 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-26T06:34:59.144 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 4 2024-03-26T06:34:59.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 -- 172.21.15.38:0/1557015238 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62bc0a2800 msgr2=0x7f62bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:59.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62bc0a2800 0x7f62bc0a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f62c002c170 tx=0x7f62c002df90 comp rx=0 tx=0).stop 2024-03-26T06:34:59.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 -- 172.21.15.38:0/1557015238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62d0101d90 msgr2=0x7f62d0197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:34:59.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62d0101d90 0x7f62d0197640 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00c970 tx=0x7f62cc00ce30 comp rx=0 tx=0).stop 2024-03-26T06:34:59.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 -- 172.21.15.38:0/1557015238 shutdown_connections 2024-03-26T06:34:59.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62bc0a2800 0x7f62bc0a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:59.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62d0106ff0 0x7f62d019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:59.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62d01046c0 0x7f62d0197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:59.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 --2- 172.21.15.38:0/1557015238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62d0101d90 0x7f62d0197640 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:34:59.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 -- 172.21.15.38:0/1557015238 >> 172.21.15.38:0/1557015238 conn(0x7f62d00fb780 msgr2=0x7f62d0102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:34:59.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 -- 172.21.15.38:0/1557015238 shutdown_connections 2024-03-26T06:34:59.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:34:59.143+0000 7f62d7fe6640 1 -- 172.21.15.38:0/1557015238 wait complete. 2024-03-26T06:34:59.534 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:34:59 smithi038 systemd[1]: Starting Ceph mds.d for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:34:59.804 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:34:59 smithi038 podman[70643]: 2024-03-26 06:34:59.532351007 +0000 UTC m=+0.098730109 container create 35bfa5bb4a686081d148620b719e162a90043494fabf6690412a7477cb1557f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T06:34:59.804 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:34:59 smithi038 podman[70643]: 2024-03-26 06:34:59.451614555 +0000 UTC m=+0.017993614 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:34:59.804 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:34:59 smithi038 podman[70643]: 2024-03-26 06:34:59.776427007 +0000 UTC m=+0.342806064 container init 35bfa5bb4a686081d148620b719e162a90043494fabf6690412a7477cb1557f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T06:34:59.804 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:34:59 smithi038 podman[70643]: 2024-03-26 06:34:59.778901991 +0000 UTC m=+0.345281048 container start 35bfa5bb4a686081d148620b719e162a90043494fabf6690412a7477cb1557f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:34:59.804 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:34:59 smithi038 ceph-mds[70663]: -- 172.21.15.38:0/3206358303 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d39ec2da40 con 0x55d39ec01000 2024-03-26T06:35:00.083 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:34:59 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-d[70659]: starting mds.d at 2024-03-26T06:35:00.083 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:34:59 smithi038 bash[70643]: 35bfa5bb4a686081d148620b719e162a90043494fabf6690412a7477cb1557f0 2024-03-26T06:35:00.083 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:34:59 smithi038 systemd[1]: Started Ceph mds.d for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:35:00.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:00 smithi121 ceph-mon[36349]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:00.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:00 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1557015238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:00.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:00 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:00.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:00 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:00.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:00 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:00.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:00 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:00.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:00 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:00.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:00 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:00.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:00 smithi170 ceph-mon[36912]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:00.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:00 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1557015238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:00.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:00 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:00.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:00 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:00.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:00 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:00.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:00 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:00.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:00 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:00.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:00 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:00 smithi038 ceph-mon[32265]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:00 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1557015238' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:00 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:00 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:00 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:00 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:00 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:00 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:01.157 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:35:01.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.274+0000 7fdd17420640 1 -- 172.21.15.38:0/1855234328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd101024f0 msgr2=0x7fdd10102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:01.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.274+0000 7fdd17420640 1 --2- 172.21.15.38:0/1855234328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd101024f0 0x7fdd10102970 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009900 tx=0x7fdd0002bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:01.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.274+0000 7fdd17420640 1 -- 172.21.15.38:0/1855234328 shutdown_connections 2024-03-26T06:35:01.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.274+0000 7fdd17420640 1 --2- 172.21.15.38:0/1855234328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd10102eb0 0x7fdd10109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:01.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.274+0000 7fdd17420640 1 --2- 172.21.15.38:0/1855234328 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd101024f0 0x7fdd10102970 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:01.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.274+0000 7fdd17420640 1 --2- 172.21.15.38:0/1855234328 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd10101300 0x7fdd10101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.274+0000 7fdd17420640 1 -- 172.21.15.38:0/1855234328 >> 172.21.15.38:0/1855234328 conn(0x7fdd100fc8d0 msgr2=0x7fdd100fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.274+0000 7fdd17420640 1 -- 172.21.15.38:0/1855234328 shutdown_connections 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.274+0000 7fdd17420640 1 -- 172.21.15.38:0/1855234328 wait complete. 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd17420640 1 Processor -- start 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd17420640 1 -- start start 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd17420640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd10101300 0x7fdd10197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd17420640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd101024f0 0x7fdd10197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd17420640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd10102eb0 0x7fdd1019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd17420640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1010cf90 con 0x7fdd101024f0 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd17420640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1010ce10 con 0x7fdd10102eb0 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd17420640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1010d110 con 0x7fdd10101300 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd149bb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd101024f0 0x7fdd10197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:01.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd149bb640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd101024f0 0x7fdd10197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58518/0 (socket says 172.21.15.38:58518) 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd149bb640 1 -- 172.21.15.38:0/1384086098 learned_addr learned my addr 172.21.15.38:0/1384086098 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd151bc640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd10101300 0x7fdd10197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd159bd640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd10102eb0 0x7fdd1019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd149bb640 1 -- 172.21.15.38:0/1384086098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd10101300 msgr2=0x7fdd10197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd149bb640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd10101300 0x7fdd10197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd149bb640 1 -- 172.21.15.38:0/1384086098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd10102eb0 msgr2=0x7fdd1019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd149bb640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd10102eb0 0x7fdd1019ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd149bb640 1 -- 172.21.15.38:0/1384086098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1019f260 con 0x7fdd101024f0 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.275+0000 7fdd149bb640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd101024f0 0x7fdd10197b80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009230 tx=0x7fdd00005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.276+0000 7fdd067fc640 1 -- 172.21.15.38:0/1384086098 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00041070 con 0x7fdd101024f0 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.276+0000 7fdd17420640 1 -- 172.21.15.38:0/1384086098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1019f4f0 con 0x7fdd101024f0 2024-03-26T06:35:01.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.276+0000 7fdd067fc640 1 -- 172.21.15.38:0/1384086098 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd000044d0 con 0x7fdd101024f0 2024-03-26T06:35:01.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.276+0000 7fdd067fc640 1 -- 172.21.15.38:0/1384086098 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd00034440 con 0x7fdd101024f0 2024-03-26T06:35:01.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.276+0000 7fdd17420640 1 -- 172.21.15.38:0/1384086098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1019fa30 con 0x7fdd101024f0 2024-03-26T06:35:01.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.277+0000 7fdd17420640 1 -- 172.21.15.38:0/1384086098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd10108670 con 0x7fdd101024f0 2024-03-26T06:35:01.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.278+0000 7fdd067fc640 1 -- 172.21.15.38:0/1384086098 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdd00044070 con 0x7fdd101024f0 2024-03-26T06:35:01.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.278+0000 7fdd067fc640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdcfc0a2800 0x7fdcfc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:01.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.279+0000 7fdd151bc640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdcfc0a2800 0x7fdcfc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:01.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.279+0000 7fdd067fc640 1 -- 172.21.15.38:0/1384086098 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdd000f05c0 con 0x7fdd101024f0 2024-03-26T06:35:01.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.279+0000 7fdd151bc640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdcfc0a2800 0x7fdcfc0a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdd08005fd0 tx=0x7fdd0800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:01.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.282+0000 7fdd067fc640 1 -- 172.21.15.38:0/1384086098 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdd000bda60 con 0x7fdd101024f0 2024-03-26T06:35:01.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:01 smithi121 ceph-mon[36349]: Deploying daemon mds.e on smithi121 2024-03-26T06:35:01.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:01 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.38:6836/3363372565,v1:172.21.15.38:6837/3363372565] up:boot 2024-03-26T06:35:01.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:01 smithi121 ceph-mon[36349]: fsmap 4 up:standby 2024-03-26T06:35:01.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:01 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-26T06:35:01.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:01 smithi170 ceph-mon[36912]: Deploying daemon mds.e on smithi121 2024-03-26T06:35:01.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:01 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.38:6836/3363372565,v1:172.21.15.38:6837/3363372565] up:boot 2024-03-26T06:35:01.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:01 smithi170 ceph-mon[36912]: fsmap 4 up:standby 2024-03-26T06:35:01.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:01 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-26T06:35:01.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.502+0000 7fdd17420640 1 -- 172.21.15.38:0/1384086098 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd1002ce70 con 0x7fdd101024f0 2024-03-26T06:35:01.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.502+0000 7fdd067fc640 1 -- 172.21.15.38:0/1384086098 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3704 (secure 0 0 0) 0x7fdd000bd360 con 0x7fdd101024f0 2024-03-26T06:35:01.505 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:01.505 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-26T06:35:01.506 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 5 2024-03-26T06:35:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.504+0000 7fdd17420640 1 -- 172.21.15.38:0/1384086098 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdcfc0a2800 msgr2=0x7fdcfc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.504+0000 7fdd17420640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdcfc0a2800 0x7fdcfc0a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdd08005fd0 tx=0x7fdd0800b390 comp rx=0 tx=0).stop 2024-03-26T06:35:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.504+0000 7fdd17420640 1 -- 172.21.15.38:0/1384086098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd101024f0 msgr2=0x7fdd10197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.504+0000 7fdd17420640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd101024f0 0x7fdd10197b80 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009230 tx=0x7fdd00005d50 comp rx=0 tx=0).stop 2024-03-26T06:35:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.505+0000 7fdd17420640 1 -- 172.21.15.38:0/1384086098 shutdown_connections 2024-03-26T06:35:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.505+0000 7fdd17420640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdcfc0a2800 0x7fdcfc0a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:01.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.505+0000 7fdd17420640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdd10102eb0 0x7fdd1019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.505+0000 7fdd17420640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdd101024f0 0x7fdd10197b80 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.505+0000 7fdd17420640 1 --2- 172.21.15.38:0/1384086098 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdd10101300 0x7fdd10197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.505+0000 7fdd17420640 1 -- 172.21.15.38:0/1384086098 >> 172.21.15.38:0/1384086098 conn(0x7fdd100fc8d0 msgr2=0x7fdd10105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.505+0000 7fdd17420640 1 -- 172.21.15.38:0/1384086098 shutdown_connections 2024-03-26T06:35:01.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:01.505+0000 7fdd17420640 1 -- 172.21.15.38:0/1384086098 wait complete. 2024-03-26T06:35:01.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:01 smithi038 ceph-mon[32265]: Deploying daemon mds.e on smithi121 2024-03-26T06:35:01.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:01 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.38:6836/3363372565,v1:172.21.15.38:6837/3363372565] up:boot 2024-03-26T06:35:01.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:01 smithi038 ceph-mon[32265]: fsmap 4 up:standby 2024-03-26T06:35:01.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:01 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-26T06:35:02.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:02 smithi121 ceph-mon[36349]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:02.295 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:02 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1384086098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:02.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:02 smithi170 ceph-mon[36912]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:02.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:02 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1384086098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:02.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:02 smithi038 ceph-mon[32265]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:02.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:02 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1384086098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:02.589 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:35:02 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-e[66183]: starting mds.e at 2024-03-26T06:35:02.895 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:35:02 smithi121 bash[66168]: 1fd8477c68b1af57e83bd3f2888ec7d1d0ebfbaa47d9e769449aea20c5835f99 2024-03-26T06:35:02.895 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:35:02 smithi121 systemd[1]: Started Ceph mds.e for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:35:03.519 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:35:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.632+0000 7f772ae73640 1 -- 172.21.15.38:0/801966915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7724101e10 msgr2=0x7f7724104200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.632+0000 7f772ae73640 1 --2- 172.21.15.38:0/801966915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7724101e10 0x7f7724104200 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7718009900 tx=0x7f771802bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.632+0000 7f772ae73640 1 -- 172.21.15.38:0/801966915 shutdown_connections 2024-03-26T06:35:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.632+0000 7f772ae73640 1 --2- 172.21.15.38:0/801966915 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7724107070 0x7f77241094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.632+0000 7f772ae73640 1 --2- 172.21.15.38:0/801966915 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7724104740 0x7f7724106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.632+0000 7f772ae73640 1 --2- 172.21.15.38:0/801966915 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7724101e10 0x7f7724104200 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.632+0000 7f772ae73640 1 -- 172.21.15.38:0/801966915 >> 172.21.15.38:0/801966915 conn(0x7f77240fb820 msgr2=0x7f77240fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.632+0000 7f772ae73640 1 -- 172.21.15.38:0/801966915 shutdown_connections 2024-03-26T06:35:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.632+0000 7f772ae73640 1 -- 172.21.15.38:0/801966915 wait complete. 2024-03-26T06:35:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f772ae73640 1 Processor -- start 2024-03-26T06:35:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f772ae73640 1 -- start start 2024-03-26T06:35:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f772ae73640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7724101e10 0x7f77241975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f772ae73640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7724104740 0x7f7724197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f772ae73640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7724107070 0x7f772419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f772ae73640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772410cfb0 con 0x7f7724104740 2024-03-26T06:35:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f772ae73640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772410ce30 con 0x7f7724107070 2024-03-26T06:35:03.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f772ae73640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772410d130 con 0x7f7724101e10 2024-03-26T06:35:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f7723fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7724104740 0x7f7724197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f7723fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7724104740 0x7f7724197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58534/0 (socket says 172.21.15.38:58534) 2024-03-26T06:35:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f7723fff640 1 -- 172.21.15.38:0/4147898736 learned_addr learned my addr 172.21.15.38:0/4147898736 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f7728c0f640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7724101e10 0x7f77241975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f7729410640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7724107070 0x7f772419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f7723fff640 1 -- 172.21.15.38:0/4147898736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7724101e10 msgr2=0x7f77241975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f7723fff640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7724101e10 0x7f77241975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:03.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f7723fff640 1 -- 172.21.15.38:0/4147898736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7724107070 msgr2=0x7f772419eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.633+0000 7f7723fff640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7724107070 0x7f772419eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.634+0000 7f7723fff640 1 -- 172.21.15.38:0/4147898736 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772419f1c0 con 0x7f7724104740 2024-03-26T06:35:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.634+0000 7f7729410640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7724107070 0x7f772419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.634+0000 7f7723fff640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7724104740 0x7f7724197ae0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f771000c970 tx=0x7f771000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.634+0000 7f7721ffb640 1 -- 172.21.15.38:0/4147898736 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7710014070 con 0x7f7724104740 2024-03-26T06:35:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.634+0000 7f772ae73640 1 -- 172.21.15.38:0/4147898736 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772419f4b0 con 0x7f7724104740 2024-03-26T06:35:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.634+0000 7f7721ffb640 1 -- 172.21.15.38:0/4147898736 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7710004440 con 0x7f7724104740 2024-03-26T06:35:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.634+0000 7f7721ffb640 1 -- 172.21.15.38:0/4147898736 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771000f450 con 0x7f7724104740 2024-03-26T06:35:03.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.634+0000 7f772ae73640 1 -- 172.21.15.38:0/4147898736 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772419fa00 con 0x7f7724104740 2024-03-26T06:35:03.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.635+0000 7f772ae73640 1 -- 172.21.15.38:0/4147898736 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77240fd310 con 0x7f7724104740 2024-03-26T06:35:03.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.636+0000 7f7721ffb640 1 -- 172.21.15.38:0/4147898736 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7710017070 con 0x7f7724104740 2024-03-26T06:35:03.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.637+0000 7f7721ffb640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f77140a2800 0x7f77140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:03.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.637+0000 7f7728c0f640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f77140a2800 0x7f77140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:03.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.637+0000 7f7721ffb640 1 -- 172.21.15.38:0/4147898736 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f77100ce410 con 0x7f7724104740 2024-03-26T06:35:03.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.637+0000 7f7728c0f640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f77140a2800 0x7f77140a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f771802c7b0 tx=0x7f7718005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:03.647 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.645+0000 7f7721ffb640 1 -- 172.21.15.38:0/4147898736 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f771001e020 con 0x7f7724104740 2024-03-26T06:35:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:03 smithi038 ceph-mon[32265]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:03.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:03.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:03.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:03.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:03.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:03.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:03 smithi038 ceph-mon[32265]: Deploying daemon mds.f on smithi170 2024-03-26T06:35:03.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:03 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.121:6834/3384748966,v1:172.21.15.121:6835/3384748966] up:boot 2024-03-26T06:35:03.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:03 smithi038 ceph-mon[32265]: fsmap 5 up:standby 2024-03-26T06:35:03.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:03 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-26T06:35:03.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.866+0000 7f772ae73640 1 -- 172.21.15.38:0/4147898736 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f772402ce70 con 0x7f7724104740 2024-03-26T06:35:03.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.866+0000 7f7721ffb640 1 -- 172.21.15.38:0/4147898736 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4498 (secure 0 0 0) 0x7f771009b950 con 0x7f7724104740 2024-03-26T06:35:03.869 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:03.869 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-26T06:35:03.869 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 6 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 -- 172.21.15.38:0/4147898736 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f77140a2800 msgr2=0x7f77140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f77140a2800 0x7f77140a4cc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f771802c7b0 tx=0x7f7718005fd0 comp rx=0 tx=0).stop 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 -- 172.21.15.38:0/4147898736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7724104740 msgr2=0x7f7724197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7724104740 0x7f7724197ae0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f771000c970 tx=0x7f771000ce30 comp rx=0 tx=0).stop 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 -- 172.21.15.38:0/4147898736 shutdown_connections 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f77140a2800 0x7f77140a4cc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7724107070 0x7f772419eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7724104740 0x7f7724197ae0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 --2- 172.21.15.38:0/4147898736 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7724101e10 0x7f77241975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 -- 172.21.15.38:0/4147898736 >> 172.21.15.38:0/4147898736 conn(0x7f77240fb820 msgr2=0x7f7724102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 -- 172.21.15.38:0/4147898736 shutdown_connections 2024-03-26T06:35:03.871 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:03.868+0000 7f772ae73640 1 -- 172.21.15.38:0/4147898736 wait complete. 2024-03-26T06:35:03.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:03 smithi121 ceph-mon[36349]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:03.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:03.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:03.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:03.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:03.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:03.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:03.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:03 smithi121 ceph-mon[36349]: Deploying daemon mds.f on smithi170 2024-03-26T06:35:03.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:03 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.121:6834/3384748966,v1:172.21.15.121:6835/3384748966] up:boot 2024-03-26T06:35:03.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:03 smithi121 ceph-mon[36349]: fsmap 5 up:standby 2024-03-26T06:35:03.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:03 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-26T06:35:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:03 smithi170 ceph-mon[36912]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:03 smithi170 ceph-mon[36912]: Deploying daemon mds.f on smithi170 2024-03-26T06:35:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:03 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.121:6834/3384748966,v1:172.21.15.121:6835/3384748966] up:boot 2024-03-26T06:35:03.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:03 smithi170 ceph-mon[36912]: fsmap 5 up:standby 2024-03-26T06:35:03.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:03 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-26T06:35:04.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:04 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/4147898736' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:04.918 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:35:04 smithi170 systemd[1]: Starting Ceph mds.f for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:35:05.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:04 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/4147898736' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:05.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:04 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/4147898736' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:05.248 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:35:04 smithi170 podman[67713]: 2024-03-26 06:35:04.884214334 +0000 UTC m=+0.018147300 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:35:05.249 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:35:04 smithi170 podman[67713]: 2024-03-26 06:35:04.99427675 +0000 UTC m=+0.128209711 container create 272d2c6aaa37a15aa5ce02dba77c314a574a523d735c5e0b8c1ba4e93f065fe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f, RELEASE=rishabh-2024mar8-2-7e98f5f, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T06:35:05.249 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:35:05 smithi170 podman[67713]: 2024-03-26 06:35:05.221542219 +0000 UTC m=+0.355475186 container init 272d2c6aaa37a15aa5ce02dba77c314a574a523d735c5e0b8c1ba4e93f065fe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-26T06:35:05.249 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:35:05 smithi170 podman[67713]: 2024-03-26 06:35:05.224196056 +0000 UTC m=+0.358129022 container start 272d2c6aaa37a15aa5ce02dba77c314a574a523d735c5e0b8c1ba4e93f065fe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T06:35:05.647 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:35:05 smithi170 ceph-mds[67732]: -- 172.21.15.170:0/2502024046 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a8d6effa40 con 0x55a8d6ed2c00 2024-03-26T06:35:05.647 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:35:05 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f[67728]: starting mds.f at 2024-03-26T06:35:05.647 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:35:05 smithi170 bash[67713]: 272d2c6aaa37a15aa5ce02dba77c314a574a523d735c5e0b8c1ba4e93f065fe7 2024-03-26T06:35:05.647 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:35:05 smithi170 systemd[1]: Started Ceph mds.f for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:35:05.882 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:35:05.909 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:05 smithi038 ceph-mon[32265]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:05.909 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:05.909 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:05.909 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:05.909 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:05.909 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:05.910 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:05 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:05.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:05 smithi170 ceph-mon[36912]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:05.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:05.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:05.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:05.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:05.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:05.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:05 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:05.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 -- 172.21.15.38:0/480224282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc580101400 msgr2=0x7fc580101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:05.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 --2- 172.21.15.38:0/480224282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc580101400 0x7fc580101880 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc570009900 tx=0x7fc57002bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:05.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 -- 172.21.15.38:0/480224282 shutdown_connections 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 --2- 172.21.15.38:0/480224282 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc580101dc0 0x7fc580108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 --2- 172.21.15.38:0/480224282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc580101400 0x7fc580101880 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 --2- 172.21.15.38:0/480224282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc580100210 0x7fc580100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 -- 172.21.15.38:0/480224282 >> 172.21.15.38:0/480224282 conn(0x7fc5800fb780 msgr2=0x7fc5800fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 -- 172.21.15.38:0/480224282 shutdown_connections 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 -- 172.21.15.38:0/480224282 wait complete. 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 Processor -- start 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 -- start start 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc580100210 0x7fc580068fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc580101400 0x7fc58019c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc580101dc0 0x7fc58019e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:05.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58010bea0 con 0x7fc580101400 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58010bd20 con 0x7fc580101dc0 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.993+0000 7fc58768c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58010c020 con 0x7fc580100210 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc584c27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc580101400 0x7fc58019c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc584c27640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc580101400 0x7fc58019c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43384/0 (socket says 172.21.15.38:43384) 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc584c27640 1 -- 172.21.15.38:0/3898279512 learned_addr learned my addr 172.21.15.38:0/3898279512 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc585428640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc580100210 0x7fc580068fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc584c27640 1 -- 172.21.15.38:0/3898279512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc580100210 msgr2=0x7fc580068fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc584c27640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc580100210 0x7fc580068fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc584c27640 1 -- 172.21.15.38:0/3898279512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc580101dc0 msgr2=0x7fc58019e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc585c29640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc580101dc0 0x7fc58019e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc584c27640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc580101dc0 0x7fc58019e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:05.999 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc584c27640 1 -- 172.21.15.38:0/3898279512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58019f2a0 con 0x7fc580101400 2024-03-26T06:35:06.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc584c27640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc580101400 0x7fc58019c370 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc570009230 tx=0x7fc570005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:06.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc5767fc640 1 -- 172.21.15.38:0/3898279512 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570041070 con 0x7fc580101400 2024-03-26T06:35:06.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc5767fc640 1 -- 172.21.15.38:0/3898279512 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5700044d0 con 0x7fc580101400 2024-03-26T06:35:06.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc58768c640 1 -- 172.21.15.38:0/3898279512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58019f4d0 con 0x7fc580101400 2024-03-26T06:35:06.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc5767fc640 1 -- 172.21.15.38:0/3898279512 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570034440 con 0x7fc580101400 2024-03-26T06:35:06.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.994+0000 7fc58768c640 1 -- 172.21.15.38:0/3898279512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58019fa10 con 0x7fc580101400 2024-03-26T06:35:06.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.995+0000 7fc58768c640 1 -- 172.21.15.38:0/3898279512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc580107530 con 0x7fc580101400 2024-03-26T06:35:06.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.996+0000 7fc5767fc640 1 -- 172.21.15.38:0/3898279512 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc570044070 con 0x7fc580101400 2024-03-26T06:35:06.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.997+0000 7fc5767fc640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc56c0a2880 0x7fc56c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:06.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.997+0000 7fc585428640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc56c0a2880 0x7fc56c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:06.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.997+0000 7fc5767fc640 1 -- 172.21.15.38:0/3898279512 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc5700f0200 con 0x7fc580101400 2024-03-26T06:35:06.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:05.997+0000 7fc585428640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc56c0a2880 0x7fc56c0a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc57c005fd0 tx=0x7fc57c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:06.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.002+0000 7fc5767fc640 1 -- 172.21.15.38:0/3898279512 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc5700bd740 con 0x7fc580101400 2024-03-26T06:35:06.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:05 smithi121 ceph-mon[36349]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:06.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:06.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:06.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:06.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:06.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:06.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:05 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:06.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.222+0000 7fc58768c640 1 -- 172.21.15.38:0/3898279512 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc58002ce70 con 0x7fc580101400 2024-03-26T06:35:06.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.223+0000 7fc5767fc640 1 -- 172.21.15.38:0/3898279512 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5292 (secure 0 0 0) 0x7fc5700bd040 con 0x7fc580101400 2024-03-26T06:35:06.225 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:06.225 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-26T06:35:06.225 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 7 2024-03-26T06:35:06.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.224+0000 7fc58768c640 1 -- 172.21.15.38:0/3898279512 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc56c0a2880 msgr2=0x7fc56c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:06.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.224+0000 7fc58768c640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc56c0a2880 0x7fc56c0a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc57c005fd0 tx=0x7fc57c00b410 comp rx=0 tx=0).stop 2024-03-26T06:35:06.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.224+0000 7fc58768c640 1 -- 172.21.15.38:0/3898279512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc580101400 msgr2=0x7fc58019c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:06.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.224+0000 7fc58768c640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc580101400 0x7fc58019c370 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc570009230 tx=0x7fc570005d50 comp rx=0 tx=0).stop 2024-03-26T06:35:06.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.224+0000 7fc58768c640 1 -- 172.21.15.38:0/3898279512 shutdown_connections 2024-03-26T06:35:06.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.224+0000 7fc58768c640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc56c0a2880 0x7fc56c0a4d40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:06.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.224+0000 7fc58768c640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc580101dc0 0x7fc58019e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:06.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.224+0000 7fc58768c640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc580101400 0x7fc58019c370 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:06.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.224+0000 7fc58768c640 1 --2- 172.21.15.38:0/3898279512 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc580100210 0x7fc580068fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:06.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.225+0000 7fc58768c640 1 -- 172.21.15.38:0/3898279512 >> 172.21.15.38:0/3898279512 conn(0x7fc5800fb780 msgr2=0x7fc580104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:06.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.225+0000 7fc58768c640 1 -- 172.21.15.38:0/3898279512 shutdown_connections 2024-03-26T06:35:06.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:06.225+0000 7fc58768c640 1 -- 172.21.15.38:0/3898279512 wait complete. 2024-03-26T06:35:06.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:06 smithi170 ceph-mon[36912]: Deploying daemon mds.c on smithi170 2024-03-26T06:35:06.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:06 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.170:6834/2374885191,v1:172.21.15.170:6835/2374885191] up:boot 2024-03-26T06:35:06.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:06 smithi170 ceph-mon[36912]: fsmap 6 up:standby 2024-03-26T06:35:06.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:06 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-26T06:35:06.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:06 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3898279512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:07.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:06 smithi038 ceph-mon[32265]: Deploying daemon mds.c on smithi170 2024-03-26T06:35:07.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:06 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.170:6834/2374885191,v1:172.21.15.170:6835/2374885191] up:boot 2024-03-26T06:35:07.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:06 smithi038 ceph-mon[32265]: fsmap 6 up:standby 2024-03-26T06:35:07.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:06 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-26T06:35:07.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:06 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3898279512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:07.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:06 smithi121 ceph-mon[36349]: Deploying daemon mds.c on smithi170 2024-03-26T06:35:07.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:06 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.170:6834/2374885191,v1:172.21.15.170:6835/2374885191] up:boot 2024-03-26T06:35:07.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:06 smithi121 ceph-mon[36349]: fsmap 6 up:standby 2024-03-26T06:35:07.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:06 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-26T06:35:07.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:06 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3898279512' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:07.658 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:35:07 smithi170 systemd[1]: Starting Ceph mds.c for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:35:07.910 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:07 smithi170 ceph-mon[36912]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:07.911 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:35:07 smithi170 podman[68019]: 2024-03-26 06:35:07.655917327 +0000 UTC m=+0.113663744 container create 33d915e9dd0a2a1202eafbf23afdf4fdfee388288f5c3005d6c1179c301d8592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-03-26T06:35:07.911 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:35:07 smithi170 podman[68019]: 2024-03-26 06:35:07.56001607 +0000 UTC m=+0.017762483 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:35:07.911 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:35:07 smithi170 podman[68019]: 2024-03-26 06:35:07.883807774 +0000 UTC m=+0.341554175 container init 33d915e9dd0a2a1202eafbf23afdf4fdfee388288f5c3005d6c1179c301d8592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T06:35:07.911 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:35:07 smithi170 podman[68019]: 2024-03-26 06:35:07.886411859 +0000 UTC m=+0.344158263 container start 33d915e9dd0a2a1202eafbf23afdf4fdfee388288f5c3005d6c1179c301d8592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-26T06:35:07.911 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:35:07 smithi170 ceph-mds[68038]: -- 172.21.15.170:0/2478614023 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557254b83a40 con 0x557254b56c00 2024-03-26T06:35:08.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:07 smithi038 ceph-mon[32265]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:08.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:07 smithi121 ceph-mon[36349]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:08.167 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:35:07 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: starting mds.c at 2024-03-26T06:35:08.168 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:35:07 smithi170 bash[68019]: 33d915e9dd0a2a1202eafbf23afdf4fdfee388288f5c3005d6c1179c301d8592 2024-03-26T06:35:08.168 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:35:07 smithi170 systemd[1]: Started Ceph mds.c for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:35:08.238 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:35:08.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.352+0000 7f503fe0b640 1 -- 172.21.15.38:0/3515395518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5038102ed0 msgr2=0x7f5038109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:08.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.352+0000 7f503fe0b640 1 --2- 172.21.15.38:0/3515395518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5038102ed0 0x7f5038109760 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5028009ca0 tx=0x7f502802c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:08.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.352+0000 7f503fe0b640 1 -- 172.21.15.38:0/3515395518 shutdown_connections 2024-03-26T06:35:08.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.352+0000 7f503fe0b640 1 --2- 172.21.15.38:0/3515395518 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5038102ed0 0x7f5038109760 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:08.357 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.352+0000 7f503fe0b640 1 --2- 172.21.15.38:0/3515395518 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5038102510 0x7f5038102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.352+0000 7f503fe0b640 1 --2- 172.21.15.38:0/3515395518 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5038101320 0x7f5038101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.352+0000 7f503fe0b640 1 -- 172.21.15.38:0/3515395518 >> 172.21.15.38:0/3515395518 conn(0x7f50380fc8d0 msgr2=0x7f50380fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.352+0000 7f503fe0b640 1 -- 172.21.15.38:0/3515395518 shutdown_connections 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.352+0000 7f503fe0b640 1 -- 172.21.15.38:0/3515395518 wait complete. 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503fe0b640 1 Processor -- start 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503fe0b640 1 -- start start 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503fe0b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5038101320 0x7f50381975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503fe0b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5038102510 0x7f5038197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503fe0b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5038102ed0 0x7f503819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503fe0b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503810cfb0 con 0x7f5038102ed0 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503fe0b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503810ce30 con 0x7f5038101320 2024-03-26T06:35:08.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503fe0b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503810d130 con 0x7f5038102510 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503e3a8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5038102ed0 0x7f503819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503e3a8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5038102ed0 0x7f503819eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43416/0 (socket says 172.21.15.38:43416) 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503e3a8640 1 -- 172.21.15.38:0/1310131951 learned_addr learned my addr 172.21.15.38:0/1310131951 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503dba7640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5038101320 0x7f50381975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503d3a6640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5038102510 0x7f5038197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503e3a8640 1 -- 172.21.15.38:0/1310131951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5038102510 msgr2=0x7f5038197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503e3a8640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5038102510 0x7f5038197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503e3a8640 1 -- 172.21.15.38:0/1310131951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5038101320 msgr2=0x7f50381975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.353+0000 7f503e3a8640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5038101320 0x7f50381975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.354+0000 7f503e3a8640 1 -- 172.21.15.38:0/1310131951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503819f1c0 con 0x7f5038102ed0 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.354+0000 7f503e3a8640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5038102ed0 0x7f503819eb50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5028037040 tx=0x7f5028032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:08.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.354+0000 7f5036ffd640 1 -- 172.21.15.38:0/1310131951 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5028041020 con 0x7f5038102ed0 2024-03-26T06:35:08.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.354+0000 7f5036ffd640 1 -- 172.21.15.38:0/1310131951 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5028034d90 con 0x7f5038102ed0 2024-03-26T06:35:08.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.354+0000 7f503fe0b640 1 -- 172.21.15.38:0/1310131951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503819f450 con 0x7f5038102ed0 2024-03-26T06:35:08.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.354+0000 7f5036ffd640 1 -- 172.21.15.38:0/1310131951 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5028039040 con 0x7f5038102ed0 2024-03-26T06:35:08.360 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.354+0000 7f503fe0b640 1 -- 172.21.15.38:0/1310131951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503819f940 con 0x7f5038102ed0 2024-03-26T06:35:08.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.355+0000 7f503fe0b640 1 -- 172.21.15.38:0/1310131951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5038108640 con 0x7f5038102ed0 2024-03-26T06:35:08.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.356+0000 7f5036ffd640 1 -- 172.21.15.38:0/1310131951 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5028044020 con 0x7f5038102ed0 2024-03-26T06:35:08.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.357+0000 7f5036ffd640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f50200a2800 0x7f50200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:08.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.357+0000 7f5036ffd640 1 -- 172.21.15.38:0/1310131951 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f50280f08b0 con 0x7f5038102ed0 2024-03-26T06:35:08.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.357+0000 7f503dba7640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f50200a2800 0x7f50200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:08.361 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.358+0000 7f503dba7640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f50200a2800 0x7f50200a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5030005fd0 tx=0x7f503000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:08.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.361+0000 7f5036ffd640 1 -- 172.21.15.38:0/1310131951 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f502804b020 con 0x7f5038102ed0 2024-03-26T06:35:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.582+0000 7f503fe0b640 1 -- 172.21.15.38:0/1310131951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f503802ce70 con 0x7f5038102ed0 2024-03-26T06:35:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.582+0000 7f5036ffd640 1 -- 172.21.15.38:0/1310131951 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5292 (secure 0 0 0) 0x7f50280bddf0 con 0x7f5038102ed0 2024-03-26T06:35:08.585 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:08.585 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-26T06:35:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 7 2024-03-26T06:35:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 -- 172.21.15.38:0/1310131951 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f50200a2800 msgr2=0x7f50200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f50200a2800 0x7f50200a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5030005fd0 tx=0x7f503000b390 comp rx=0 tx=0).stop 2024-03-26T06:35:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 -- 172.21.15.38:0/1310131951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5038102ed0 msgr2=0x7f503819eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5038102ed0 0x7f503819eb50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5028037040 tx=0x7f5028032f20 comp rx=0 tx=0).stop 2024-03-26T06:35:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 -- 172.21.15.38:0/1310131951 shutdown_connections 2024-03-26T06:35:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f50200a2800 0x7f50200a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5038102ed0 0x7f503819eb50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5038102510 0x7f5038197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 --2- 172.21.15.38:0/1310131951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5038101320 0x7f50381975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 -- 172.21.15.38:0/1310131951 >> 172.21.15.38:0/1310131951 conn(0x7f50380fc8d0 msgr2=0x7f5038105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 -- 172.21.15.38:0/1310131951 shutdown_connections 2024-03-26T06:35:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:08.584+0000 7f503fe0b640 1 -- 172.21.15.38:0/1310131951 wait complete. 2024-03-26T06:35:09.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:09.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:09.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:09.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:09.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:09.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:09.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:08 smithi038 ceph-mon[32265]: Deploying daemon mds.h on smithi121 2024-03-26T06:35:09.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:08 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1310131951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:09.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:08 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.170:6836/2484220351,v1:172.21.15.170:6837/2484220351] up:boot 2024-03-26T06:35:09.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:08 smithi038 ceph-mon[32265]: fsmap 7 up:standby 2024-03-26T06:35:09.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:08 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:35:09.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:09.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:09.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:09.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:09.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:09.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:09.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:08 smithi121 ceph-mon[36349]: Deploying daemon mds.h on smithi121 2024-03-26T06:35:09.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:08 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1310131951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:09.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:08 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.170:6836/2484220351,v1:172.21.15.170:6837/2484220351] up:boot 2024-03-26T06:35:09.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:08 smithi121 ceph-mon[36349]: fsmap 7 up:standby 2024-03-26T06:35:09.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:08 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:35:09.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:09.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:09.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:09.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:09.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:09.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:09.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:08 smithi170 ceph-mon[36912]: Deploying daemon mds.h on smithi121 2024-03-26T06:35:09.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:08 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1310131951' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:09.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:08 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.170:6836/2484220351,v1:172.21.15.170:6837/2484220351] up:boot 2024-03-26T06:35:09.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:08 smithi170 ceph-mon[36912]: fsmap 7 up:standby 2024-03-26T06:35:09.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:08 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:35:10.261 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:10 smithi170 ceph-mon[36912]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:10.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:10 smithi038 ceph-mon[32265]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:10.352 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:10 smithi121 ceph-mon[36349]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:10.352 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:35:10 smithi121 systemd[1]: Starting Ceph mds.h for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:35:10.352 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:35:10 smithi121 podman[66483]: 2024-03-26 06:35:10.252633992 +0000 UTC m=+0.017723375 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:35:10.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:35:10.603 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:35:10 smithi121 podman[66483]: 2024-03-26 06:35:10.362752131 +0000 UTC m=+0.127841506 container create d6d6c8cc8bc586b700a5c1831b912d7988d2d620276141cf9f7cfcb903aae39e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:35:10.603 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:35:10 smithi121 podman[66483]: 2024-03-26 06:35:10.598682697 +0000 UTC m=+0.363772072 container init d6d6c8cc8bc586b700a5c1831b912d7988d2d620276141cf9f7cfcb903aae39e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-26T06:35:10.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.710+0000 7fac3477e640 1 -- 172.21.15.38:0/789238120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c101310 msgr2=0x7fac2c101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:10.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.710+0000 7fac3477e640 1 --2- 172.21.15.38:0/789238120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c101310 0x7fac2c101730 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009ca0 tx=0x7fac1c02c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:10.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 -- 172.21.15.38:0/789238120 shutdown_connections 2024-03-26T06:35:10.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 --2- 172.21.15.38:0/789238120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac2c102ec0 0x7fac2c109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:10.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 --2- 172.21.15.38:0/789238120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac2c102500 0x7fac2c102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:10.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 --2- 172.21.15.38:0/789238120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c101310 0x7fac2c101730 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:10.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 -- 172.21.15.38:0/789238120 >> 172.21.15.38:0/789238120 conn(0x7fac2c0fc8a0 msgr2=0x7fac2c0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:10.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 -- 172.21.15.38:0/789238120 shutdown_connections 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 -- 172.21.15.38:0/789238120 wait complete. 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 Processor -- start 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 -- start start 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac2c101310 0x7fac2c197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c102500 0x7fac2c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac2c102ec0 0x7fac2c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c10cfa0 con 0x7fac2c102500 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c10ce20 con 0x7fac2c102ec0 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac3477e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c10d120 con 0x7fac2c101310 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac31d19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c102500 0x7fac2c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac31d19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c102500 0x7fac2c197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43430/0 (socket says 172.21.15.38:43430) 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.711+0000 7fac31d19640 1 -- 172.21.15.38:0/2519375616 learned_addr learned my addr 172.21.15.38:0/2519375616 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac3251a640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac2c101310 0x7fac2c197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac32d1b640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac2c102ec0 0x7fac2c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac31d19640 1 -- 172.21.15.38:0/2519375616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac2c101310 msgr2=0x7fac2c197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac31d19640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac2c101310 0x7fac2c197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac31d19640 1 -- 172.21.15.38:0/2519375616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac2c102ec0 msgr2=0x7fac2c19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac31d19640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac2c102ec0 0x7fac2c19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac31d19640 1 -- 172.21.15.38:0/2519375616 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c19f250 con 0x7fac2c102500 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac31d19640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c102500 0x7fac2c197b70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fac2000e9f0 tx=0x7fac2000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac2b7fe640 1 -- 172.21.15.38:0/2519375616 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac2000cd80 con 0x7fac2c102500 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac2b7fe640 1 -- 172.21.15.38:0/2519375616 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac20005500 con 0x7fac2c102500 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac3477e640 1 -- 172.21.15.38:0/2519375616 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2c19f540 con 0x7fac2c102500 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac2b7fe640 1 -- 172.21.15.38:0/2519375616 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac20010690 con 0x7fac2c102500 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.712+0000 7fac3477e640 1 -- 172.21.15.38:0/2519375616 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2c19fa40 con 0x7fac2c102500 2024-03-26T06:35:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.713+0000 7fac3477e640 1 -- 172.21.15.38:0/2519375616 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2c108630 con 0x7fac2c102500 2024-03-26T06:35:10.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.714+0000 7fac2b7fe640 1 -- 172.21.15.38:0/2519375616 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fac200046c0 con 0x7fac2c102500 2024-03-26T06:35:10.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.715+0000 7fac2b7fe640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fac140a2800 0x7fac140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:10.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.715+0000 7fac3251a640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fac140a2800 0x7fac140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:10.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.715+0000 7fac2b7fe640 1 -- 172.21.15.38:0/2519375616 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fac200cea50 con 0x7fac2c102500 2024-03-26T06:35:10.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.715+0000 7fac3251a640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fac140a2800 0x7fac140a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fac1c02c700 tx=0x7fac1c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:10.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.720+0000 7fac2b7fe640 1 -- 172.21.15.38:0/2519375616 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fac2009bef0 con 0x7fac2c102500 2024-03-26T06:35:10.895 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:35:10 smithi121 podman[66483]: 2024-03-26 06:35:10.601601468 +0000 UTC m=+0.366690846 container start d6d6c8cc8bc586b700a5c1831b912d7988d2d620276141cf9f7cfcb903aae39e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T06:35:10.895 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:35:10 smithi121 ceph-mds[66502]: -- 172.21.15.121:0/2821356923 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558ba1b1ba40 con 0x558ba1aee800 2024-03-26T06:35:10.895 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:35:10 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h[66498]: starting mds.h at 2024-03-26T06:35:10.895 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:35:10 smithi121 bash[66483]: d6d6c8cc8bc586b700a5c1831b912d7988d2d620276141cf9f7cfcb903aae39e 2024-03-26T06:35:10.895 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:35:10 smithi121 systemd[1]: Started Ceph mds.h for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:35:10.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.946+0000 7fac3477e640 1 -- 172.21.15.38:0/2519375616 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac2c02ce70 con 0x7fac2c102500 2024-03-26T06:35:10.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.946+0000 7fac2b7fe640 1 -- 172.21.15.38:0/2519375616 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6086 (secure 0 0 0) 0x7fac2009b7f0 con 0x7fac2c102500 2024-03-26T06:35:10.949 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:10.949 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-26T06:35:10.949 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 8 2024-03-26T06:35:10.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 -- 172.21.15.38:0/2519375616 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fac140a2800 msgr2=0x7fac140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:10.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fac140a2800 0x7fac140a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fac1c02c700 tx=0x7fac1c032590 comp rx=0 tx=0).stop 2024-03-26T06:35:10.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 -- 172.21.15.38:0/2519375616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c102500 msgr2=0x7fac2c197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:10.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c102500 0x7fac2c197b70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fac2000e9f0 tx=0x7fac2000edf0 comp rx=0 tx=0).stop 2024-03-26T06:35:10.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 -- 172.21.15.38:0/2519375616 shutdown_connections 2024-03-26T06:35:10.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fac140a2800 0x7fac140a4cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:10.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac2c102ec0 0x7fac2c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:10.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fac2c102500 0x7fac2c197b70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:10.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 --2- 172.21.15.38:0/2519375616 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fac2c101310 0x7fac2c197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:10.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 -- 172.21.15.38:0/2519375616 >> 172.21.15.38:0/2519375616 conn(0x7fac2c0fc8a0 msgr2=0x7fac2c105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:10.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 -- 172.21.15.38:0/2519375616 shutdown_connections 2024-03-26T06:35:10.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:10.948+0000 7fac3477e640 1 -- 172.21.15.38:0/2519375616 wait complete. 2024-03-26T06:35:12.025 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:12.025 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:12.025 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:12.025 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:12.025 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:12.025 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:12.025 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:12.026 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: Deploying daemon mds.i on smithi170 2024-03-26T06:35:12.026 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2519375616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:12.026 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] up:boot 2024-03-26T06:35:12.026 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: fsmap 8 up:standby 2024-03-26T06:35:12.026 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:11 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-26T06:35:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:12.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:12.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:12.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: Deploying daemon mds.i on smithi170 2024-03-26T06:35:12.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2519375616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:12.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] up:boot 2024-03-26T06:35:12.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: fsmap 8 up:standby 2024-03-26T06:35:12.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:11 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-26T06:35:12.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:12.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:12.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:12.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:12.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:12.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:12.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:12.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: Deploying daemon mds.i on smithi170 2024-03-26T06:35:12.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2519375616' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:12.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] up:boot 2024-03-26T06:35:12.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: fsmap 8 up:standby 2024-03-26T06:35:12.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:11 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-26T06:35:12.917 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:35:12 smithi170 systemd[1]: Starting Ceph mds.i for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:35:12.962 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:35:13.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.075+0000 7f8f728b2640 1 -- 172.21.15.38:0/332943934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f6c100230 msgr2=0x7f8f6c100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:13.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.075+0000 7f8f728b2640 1 --2- 172.21.15.38:0/332943934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f6c100230 0x7f8f6c100650 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c009900 tx=0x7f8f5c02bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:13.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.075+0000 7f8f728b2640 1 -- 172.21.15.38:0/332943934 shutdown_connections 2024-03-26T06:35:13.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.075+0000 7f8f728b2640 1 --2- 172.21.15.38:0/332943934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6c101de0 0x7f8f6c108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.075+0000 7f8f728b2640 1 --2- 172.21.15.38:0/332943934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f6c101420 0x7f8f6c1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.075+0000 7f8f728b2640 1 --2- 172.21.15.38:0/332943934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f6c100230 0x7f8f6c100650 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.075+0000 7f8f728b2640 1 -- 172.21.15.38:0/332943934 >> 172.21.15.38:0/332943934 conn(0x7f8f6c0fb7a0 msgr2=0x7f8f6c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.075+0000 7f8f728b2640 1 -- 172.21.15.38:0/332943934 shutdown_connections 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.076+0000 7f8f728b2640 1 -- 172.21.15.38:0/332943934 wait complete. 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.076+0000 7f8f728b2640 1 Processor -- start 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.076+0000 7f8f728b2640 1 -- start start 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.076+0000 7f8f728b2640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6c100230 0x7f8f6c06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.076+0000 7f8f728b2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f6c101420 0x7f8f6c19c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.076+0000 7f8f728b2640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f6c101de0 0x7f8f6c19e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.076+0000 7f8f728b2640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c10be90 con 0x7f8f6c101420 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.076+0000 7f8f728b2640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c10bd10 con 0x7f8f6c100230 2024-03-26T06:35:13.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f728b2640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c10c010 con 0x7f8f6c101de0 2024-03-26T06:35:13.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f6b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f6c101420 0x7f8f6c19c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:13.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f6b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f6c101420 0x7f8f6c19c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43454/0 (socket says 172.21.15.38:43454) 2024-03-26T06:35:13.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f6bfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6c100230 0x7f8f6c06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:13.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f6b7fe640 1 -- 172.21.15.38:0/1503416133 learned_addr learned my addr 172.21.15.38:0/1503416133 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:13.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f70e4f640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f6c101de0 0x7f8f6c19e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:13.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f6b7fe640 1 -- 172.21.15.38:0/1503416133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f6c101de0 msgr2=0x7f8f6c19e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:13.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f6b7fe640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f6c101de0 0x7f8f6c19e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:13.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f6b7fe640 1 -- 172.21.15.38:0/1503416133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6c100230 msgr2=0x7f8f6c06a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:13.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f6b7fe640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6c100230 0x7f8f6c06a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:13.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f6b7fe640 1 -- 172.21.15.38:0/1503416133 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c19f280 con 0x7f8f6c101420 2024-03-26T06:35:13.082 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f6bfff640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6c100230 0x7f8f6c06a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:35:13.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f6b7fe640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f6c101420 0x7f8f6c19c380 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000ea60 tx=0x7f8f6000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:13.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f697fa640 1 -- 172.21.15.38:0/1503416133 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6000cd20 con 0x7f8f6c101420 2024-03-26T06:35:13.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f697fa640 1 -- 172.21.15.38:0/1503416133 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f600035f0 con 0x7f8f6c101420 2024-03-26T06:35:13.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f728b2640 1 -- 172.21.15.38:0/1503416133 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c19f570 con 0x7f8f6c101420 2024-03-26T06:35:13.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f697fa640 1 -- 172.21.15.38:0/1503416133 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f60010690 con 0x7f8f6c101420 2024-03-26T06:35:13.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.077+0000 7f8f728b2640 1 -- 172.21.15.38:0/1503416133 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c19fac0 con 0x7f8f6c101420 2024-03-26T06:35:13.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.078+0000 7f8f728b2640 1 -- 172.21.15.38:0/1503416133 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c107550 con 0x7f8f6c101420 2024-03-26T06:35:13.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.080+0000 7f8f697fa640 1 -- 172.21.15.38:0/1503416133 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8f600037a0 con 0x7f8f6c101420 2024-03-26T06:35:13.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.080+0000 7f8f697fa640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8f540a2800 0x7f8f540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:13.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.080+0000 7f8f6bfff640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8f540a2800 0x7f8f540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:13.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.080+0000 7f8f697fa640 1 -- 172.21.15.38:0/1503416133 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8f600ce980 con 0x7f8f6c101420 2024-03-26T06:35:13.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.081+0000 7f8f6bfff640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8f540a2800 0x7f8f540a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c000c00 tx=0x7f8f5c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:13.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.084+0000 7f8f697fa640 1 -- 172.21.15.38:0/1503416133 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8f6009bf50 con 0x7f8f6c101420 2024-03-26T06:35:13.305 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:35:13 smithi170 podman[68338]: 2024-03-26 06:35:12.944427502 +0000 UTC m=+0.017794651 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:35:13.305 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:35:13 smithi170 podman[68338]: 2024-03-26 06:35:13.074164152 +0000 UTC m=+0.147531309 container create fea61e049736062c4d0d32141be4d3c41746f55d1bf342c5bad55cd8700bf7dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T06:35:13.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.305+0000 7f8f728b2640 1 -- 172.21.15.38:0/1503416133 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f6c02ce70 con 0x7f8f6c101420 2024-03-26T06:35:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.305+0000 7f8f697fa640 1 -- 172.21.15.38:0/1503416133 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6880 (secure 0 0 0) 0x7f8f6009b850 con 0x7f8f6c101420 2024-03-26T06:35:13.308 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:13.308 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-26T06:35:13.308 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 9 2024-03-26T06:35:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 -- 172.21.15.38:0/1503416133 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8f540a2800 msgr2=0x7f8f540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8f540a2800 0x7f8f540a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c000c00 tx=0x7f8f5c005fd0 comp rx=0 tx=0).stop 2024-03-26T06:35:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 -- 172.21.15.38:0/1503416133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f6c101420 msgr2=0x7f8f6c19c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:13.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f6c101420 0x7f8f6c19c380 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8f6000ea60 tx=0x7f8f6000ee60 comp rx=0 tx=0).stop 2024-03-26T06:35:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 -- 172.21.15.38:0/1503416133 shutdown_connections 2024-03-26T06:35:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f8f540a2800 0x7f8f540a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8f6c101de0 0x7f8f6c19e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f6c101420 0x7f8f6c19c380 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 --2- 172.21.15.38:0/1503416133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f6c100230 0x7f8f6c06a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 -- 172.21.15.38:0/1503416133 >> 172.21.15.38:0/1503416133 conn(0x7f8f6c0fb7a0 msgr2=0x7f8f6c104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 -- 172.21.15.38:0/1503416133 shutdown_connections 2024-03-26T06:35:13.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:13.308+0000 7f8f728b2640 1 -- 172.21.15.38:0/1503416133 wait complete. 2024-03-26T06:35:13.667 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:35:13 smithi170 podman[68338]: 2024-03-26 06:35:13.303273852 +0000 UTC m=+0.376641007 container init fea61e049736062c4d0d32141be4d3c41746f55d1bf342c5bad55cd8700bf7dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.vendor=CentOS) 2024-03-26T06:35:13.668 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:35:13 smithi170 podman[68338]: 2024-03-26 06:35:13.306029253 +0000 UTC m=+0.379396401 container start fea61e049736062c4d0d32141be4d3c41746f55d1bf342c5bad55cd8700bf7dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:35:13.668 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:35:13 smithi170 ceph-mds[68357]: -- 172.21.15.170:0/1656635063 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ff6b823a40 con 0x55ff6b7f6800 2024-03-26T06:35:13.668 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:35:13 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i[68353]: starting mds.i at 2024-03-26T06:35:13.668 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:35:13 smithi170 bash[68338]: fea61e049736062c4d0d32141be4d3c41746f55d1bf342c5bad55cd8700bf7dd 2024-03-26T06:35:13.668 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:35:13 smithi170 systemd[1]: Started Ceph mds.i for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:35:14.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:13 smithi038 ceph-mon[32265]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:14.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:13 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1503416133' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:14.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:14.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:14.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:14.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:14.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:14.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:13 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:13 smithi121 ceph-mon[36349]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:13 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1503416133' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:14.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:13 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:14.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:13 smithi170 ceph-mon[36912]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:14.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:13 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1503416133' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:14.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:14.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:14.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:14.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:14.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:14.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:13 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:15.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:14 smithi038 ceph-mon[32265]: Deploying daemon mds.g on smithi038 2024-03-26T06:35:15.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:14 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] up:boot 2024-03-26T06:35:15.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:14 smithi038 ceph-mon[32265]: fsmap 9 up:standby 2024-03-26T06:35:15.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:14 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-26T06:35:15.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:14 smithi121 ceph-mon[36349]: Deploying daemon mds.g on smithi038 2024-03-26T06:35:15.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:14 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] up:boot 2024-03-26T06:35:15.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:14 smithi121 ceph-mon[36349]: fsmap 9 up:standby 2024-03-26T06:35:15.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:14 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-26T06:35:15.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:14 smithi170 ceph-mon[36912]: Deploying daemon mds.g on smithi038 2024-03-26T06:35:15.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:14 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] up:boot 2024-03-26T06:35:15.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:14 smithi170 ceph-mon[36912]: fsmap 9 up:standby 2024-03-26T06:35:15.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:14 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-26T06:35:15.322 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:35:15.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 -- 172.21.15.38:0/174794408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc101df0 msgr2=0x7fd0cc1041e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:15.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 --2- 172.21.15.38:0/174794408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc101df0 0x7fd0cc1041e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc009900 tx=0x7fd0bc02bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:15.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 -- 172.21.15.38:0/174794408 shutdown_connections 2024-03-26T06:35:15.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 --2- 172.21.15.38:0/174794408 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0cc107050 0x7fd0cc109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:15.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 --2- 172.21.15.38:0/174794408 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc104720 0x7fd0cc106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:15.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 --2- 172.21.15.38:0/174794408 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc101df0 0x7fd0cc1041e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:15.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 -- 172.21.15.38:0/174794408 >> 172.21.15.38:0/174794408 conn(0x7fd0cc0fb780 msgr2=0x7fd0cc0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:15.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 -- 172.21.15.38:0/174794408 shutdown_connections 2024-03-26T06:35:15.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 -- 172.21.15.38:0/174794408 wait complete. 2024-03-26T06:35:15.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 Processor -- start 2024-03-26T06:35:15.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 -- start start 2024-03-26T06:35:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc101df0 0x7fd0cc101760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc104720 0x7fd0cc0ffda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0cc107050 0x7fd0cc1002e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc10cfa0 con 0x7fd0cc104720 2024-03-26T06:35:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc10ce20 con 0x7fd0cc107050 2024-03-26T06:35:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.584+0000 7fd0d364b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0cc10d120 con 0x7fd0cc101df0 2024-03-26T06:35:15.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d0be6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc104720 0x7fd0cc0ffda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:15.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d13e7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc101df0 0x7fd0cc101760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:15.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d0be6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc104720 0x7fd0cc0ffda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33630/0 (socket says 172.21.15.38:33630) 2024-03-26T06:35:15.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d0be6640 1 -- 172.21.15.38:0/947251087 learned_addr learned my addr 172.21.15.38:0/947251087 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:15.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d1be8640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0cc107050 0x7fd0cc1002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:15.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d0be6640 1 -- 172.21.15.38:0/947251087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc101df0 msgr2=0x7fd0cc101760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:15.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d0be6640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc101df0 0x7fd0cc101760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:15.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d0be6640 1 -- 172.21.15.38:0/947251087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0cc107050 msgr2=0x7fd0cc1002e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:15.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d0be6640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0cc107050 0x7fd0cc1002e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:15.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d0be6640 1 -- 172.21.15.38:0/947251087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0cc100b70 con 0x7fd0cc104720 2024-03-26T06:35:15.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d0be6640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc104720 0x7fd0cc0ffda0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800d770 tx=0x7fd0c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:15.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0c27fc640 1 -- 172.21.15.38:0/947251087 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c8017070 con 0x7fd0cc104720 2024-03-26T06:35:15.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0c27fc640 1 -- 172.21.15.38:0/947251087 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0c8009c30 con 0x7fd0cc104720 2024-03-26T06:35:15.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d364b640 1 -- 172.21.15.38:0/947251087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0cc1a1780 con 0x7fd0cc104720 2024-03-26T06:35:15.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0c27fc640 1 -- 172.21.15.38:0/947251087 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c800f620 con 0x7fd0cc104720 2024-03-26T06:35:15.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.585+0000 7fd0d364b640 1 -- 172.21.15.38:0/947251087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0cc1a1cd0 con 0x7fd0cc104720 2024-03-26T06:35:15.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.587+0000 7fd0d364b640 1 -- 172.21.15.38:0/947251087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0cc0fd270 con 0x7fd0cc104720 2024-03-26T06:35:15.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.588+0000 7fd0c27fc640 1 -- 172.21.15.38:0/947251087 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd0c801a070 con 0x7fd0cc104720 2024-03-26T06:35:15.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.589+0000 7fd0c27fc640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd0b80a2800 0x7fd0b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:15.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.589+0000 7fd0d13e7640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd0b80a2800 0x7fd0b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:15.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.589+0000 7fd0c27fc640 1 -- 172.21.15.38:0/947251087 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd0c80ce080 con 0x7fd0cc104720 2024-03-26T06:35:15.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.589+0000 7fd0d13e7640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd0b80a2800 0x7fd0b80a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc02c7b0 tx=0x7fd0bc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:15.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.594+0000 7fd0c27fc640 1 -- 172.21.15.38:0/947251087 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd0c8009de0 con 0x7fd0cc104720 2024-03-26T06:35:15.767 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:35:15 smithi038 systemd[1]: Starting Ceph mds.g for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:35:15.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.828+0000 7fd0d364b640 1 -- 172.21.15.38:0/947251087 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0cc02ce70 con 0x7fd0cc104720 2024-03-26T06:35:15.831 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.829+0000 7fd0c27fc640 1 -- 172.21.15.38:0/947251087 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7676 (secure 0 0 0) 0x7fd0c809aed0 con 0x7fd0cc104720 2024-03-26T06:35:15.831 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:15.831 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24502,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-26T06:35:15.832 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 10 2024-03-26T06:35:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 -- 172.21.15.38:0/947251087 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd0b80a2800 msgr2=0x7fd0b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd0b80a2800 0x7fd0b80a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc02c7b0 tx=0x7fd0bc005fd0 comp rx=0 tx=0).stop 2024-03-26T06:35:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 -- 172.21.15.38:0/947251087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc104720 msgr2=0x7fd0cc0ffda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc104720 0x7fd0cc0ffda0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd0c800d770 tx=0x7fd0c800dc30 comp rx=0 tx=0).stop 2024-03-26T06:35:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 -- 172.21.15.38:0/947251087 shutdown_connections 2024-03-26T06:35:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd0b80a2800 0x7fd0b80a4cc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0cc107050 0x7fd0cc1002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0cc104720 0x7fd0cc0ffda0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 --2- 172.21.15.38:0/947251087 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0cc101df0 0x7fd0cc101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 -- 172.21.15.38:0/947251087 >> 172.21.15.38:0/947251087 conn(0x7fd0cc0fb780 msgr2=0x7fd0cc102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 -- 172.21.15.38:0/947251087 shutdown_connections 2024-03-26T06:35:15.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:15.831+0000 7fd0d364b640 1 -- 172.21.15.38:0/947251087 wait complete. 2024-03-26T06:35:16.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:15 smithi038 ceph-mon[32265]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:16.081 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:35:15 smithi038 podman[71241]: 2024-03-26 06:35:15.843274625 +0000 UTC m=+0.018252745 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:35:16.081 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:35:15 smithi038 podman[71241]: 2024-03-26 06:35:15.96500525 +0000 UTC m=+0.139983393 container create 007d8530a19ef5a86c23bfdbdbdf0529662e96071e29bddbd325aad490a05005 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-g, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-26T06:35:16.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:15 smithi121 ceph-mon[36349]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:16.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:15 smithi170 ceph-mon[36912]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:16.333 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:35:16 smithi038 podman[71241]: 2024-03-26 06:35:16.225787411 +0000 UTC m=+0.400765514 container init 007d8530a19ef5a86c23bfdbdbdf0529662e96071e29bddbd325aad490a05005 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T06:35:16.334 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:35:16 smithi038 podman[71241]: 2024-03-26 06:35:16.228489443 +0000 UTC m=+0.403467555 container start 007d8530a19ef5a86c23bfdbdbdf0529662e96071e29bddbd325aad490a05005 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-g, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T06:35:16.334 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:35:16 smithi038 ceph-mds[71263]: -- 172.21.15.38:0/832962944 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55aa441bda40 con 0x55aa44191000 2024-03-26T06:35:16.334 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:35:16 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-g[71259]: starting mds.g at 2024-03-26T06:35:16.334 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:35:16 smithi038 bash[71241]: 007d8530a19ef5a86c23bfdbdbdf0529662e96071e29bddbd325aad490a05005 2024-03-26T06:35:16.334 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:35:16 smithi038 systemd[1]: Started Ceph mds.g for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:35:17.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:16 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/947251087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:17.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:16 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:17.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:16 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:17.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:16 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:17.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:16 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:17.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:16 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:17.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:16 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:17.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:16 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/947251087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:17.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:16 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:17.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:16 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:17.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:16 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:17.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:16 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:17.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:16 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:17.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:16 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:17.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:16 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/947251087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:17.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:16 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:17.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:16 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:17.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:16 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:17.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:16 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:17.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:16 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:17.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:16 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:17.846 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:35:17.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.952+0000 7f6e1229a640 1 -- 172.21.15.38:0/1120591394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c101400 msgr2=0x7f6e0c101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:17.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.952+0000 7f6e1229a640 1 --2- 172.21.15.38:0/1120591394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c101400 0x7f6e0c101880 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009900 tx=0x7f6df402bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.953+0000 7f6e1229a640 1 -- 172.21.15.38:0/1120591394 shutdown_connections 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.953+0000 7f6e1229a640 1 --2- 172.21.15.38:0/1120591394 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e0c101dc0 0x7f6e0c108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.953+0000 7f6e1229a640 1 --2- 172.21.15.38:0/1120591394 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c101400 0x7f6e0c101880 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.953+0000 7f6e1229a640 1 --2- 172.21.15.38:0/1120591394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e0c100210 0x7f6e0c100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.953+0000 7f6e1229a640 1 -- 172.21.15.38:0/1120591394 >> 172.21.15.38:0/1120591394 conn(0x7f6e0c0fb780 msgr2=0x7f6e0c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.953+0000 7f6e1229a640 1 -- 172.21.15.38:0/1120591394 shutdown_connections 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.953+0000 7f6e1229a640 1 -- 172.21.15.38:0/1120591394 wait complete. 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.953+0000 7f6e1229a640 1 Processor -- start 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.953+0000 7f6e1229a640 1 -- start start 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e1229a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c100210 0x7f6e0c199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e1229a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e0c101400 0x7f6e0c199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:17.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e1229a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e0c101dc0 0x7f6e0c1a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:17.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e1229a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c10bee0 con 0x7f6e0c100210 2024-03-26T06:35:17.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e1229a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c10bd60 con 0x7f6e0c101400 2024-03-26T06:35:17.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e1229a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0c10c060 con 0x7f6e0c101dc0 2024-03-26T06:35:17.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e0bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c100210 0x7f6e0c199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:17.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e0bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c100210 0x7f6e0c199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33658/0 (socket says 172.21.15.38:33658) 2024-03-26T06:35:17.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e0bfff640 1 -- 172.21.15.38:0/1720436730 learned_addr learned my addr 172.21.15.38:0/1720436730 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:17.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e10837640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e0c101dc0 0x7f6e0c1a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:17.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e0b7fe640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e0c101400 0x7f6e0c199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:17.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e0bfff640 1 -- 172.21.15.38:0/1720436730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e0c101dc0 msgr2=0x7f6e0c1a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:17.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e0bfff640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e0c101dc0 0x7f6e0c1a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:17.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e0bfff640 1 -- 172.21.15.38:0/1720436730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e0c101400 msgr2=0x7f6e0c199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:17.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e0bfff640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e0c101400 0x7f6e0c199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:17.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e0bfff640 1 -- 172.21.15.38:0/1720436730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0c1a1530 con 0x7f6e0c100210 2024-03-26T06:35:17.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e0bfff640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c100210 0x7f6e0c199910 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00d770 tx=0x7f6dfc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:17.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e097fa640 1 -- 172.21.15.38:0/1720436730 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc017070 con 0x7f6e0c100210 2024-03-26T06:35:17.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e097fa640 1 -- 172.21.15.38:0/1720436730 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6dfc009c30 con 0x7f6e0c100210 2024-03-26T06:35:17.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e1229a640 1 -- 172.21.15.38:0/1720436730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0c1a1820 con 0x7f6e0c100210 2024-03-26T06:35:17.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.954+0000 7f6e097fa640 1 -- 172.21.15.38:0/1720436730 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc003dd0 con 0x7f6e0c100210 2024-03-26T06:35:17.960 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.955+0000 7f6e1229a640 1 -- 172.21.15.38:0/1720436730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e0c1a1d40 con 0x7f6e0c100210 2024-03-26T06:35:17.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.955+0000 7f6e1229a640 1 -- 172.21.15.38:0/1720436730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0c1a1fa0 con 0x7f6e0c100210 2024-03-26T06:35:17.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.956+0000 7f6e097fa640 1 -- 172.21.15.38:0/1720436730 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6dfc003fb0 con 0x7f6e0c100210 2024-03-26T06:35:17.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.957+0000 7f6e097fa640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6df80a2800 0x7f6df80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:17.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.957+0000 7f6e0b7fe640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6df80a2800 0x7f6df80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:17.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.957+0000 7f6e097fa640 1 -- 172.21.15.38:0/1720436730 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6dfc0cde90 con 0x7f6e0c100210 2024-03-26T06:35:17.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.958+0000 7f6e0b7fe640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6df80a2800 0x7f6df80a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009230 tx=0x7f6df4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:17.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:17.962+0000 7f6e097fa640 1 -- 172.21.15.38:0/1720436730 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6dfc009de0 con 0x7f6e0c100210 2024-03-26T06:35:18.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:17 smithi038 ceph-mon[32265]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:18.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:17 smithi038 ceph-mon[32265]: Deploying daemon mds.a on smithi038 2024-03-26T06:35:18.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:17 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.38:6838/357411663,v1:172.21.15.38:6839/357411663] up:boot 2024-03-26T06:35:18.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:17 smithi038 ceph-mon[32265]: fsmap 10 up:standby 2024-03-26T06:35:18.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:17 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-26T06:35:18.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:17 smithi121 ceph-mon[36349]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:18.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:17 smithi121 ceph-mon[36349]: Deploying daemon mds.a on smithi038 2024-03-26T06:35:18.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:17 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.38:6838/357411663,v1:172.21.15.38:6839/357411663] up:boot 2024-03-26T06:35:18.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:17 smithi121 ceph-mon[36349]: fsmap 10 up:standby 2024-03-26T06:35:18.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:17 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-26T06:35:18.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:17 smithi170 ceph-mon[36912]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:18.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:17 smithi170 ceph-mon[36912]: Deploying daemon mds.a on smithi038 2024-03-26T06:35:18.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:17 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.38:6838/357411663,v1:172.21.15.38:6839/357411663] up:boot 2024-03-26T06:35:18.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:17 smithi170 ceph-mon[36912]: fsmap 10 up:standby 2024-03-26T06:35:18.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:17 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-26T06:35:18.189 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.187+0000 7f6e1229a640 1 -- 172.21.15.38:0/1720436730 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e0c1a2170 con 0x7f6e0c100210 2024-03-26T06:35:18.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.188+0000 7f6e097fa640 1 -- 172.21.15.38:0/1720436730 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8465 (secure 0 0 0) 0x7f6dfc09ad80 con 0x7f6e0c100210 2024-03-26T06:35:18.190 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:18.190 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/357411663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":357411663},{"type":"v1","addr":"172.21.15.38:6839","nonce":357411663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24502,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-26T06:35:18.191 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 11 2024-03-26T06:35:18.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 -- 172.21.15.38:0/1720436730 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6df80a2800 msgr2=0x7f6df80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:18.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6df80a2800 0x7f6df80a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6df4009230 tx=0x7f6df4005fd0 comp rx=0 tx=0).stop 2024-03-26T06:35:18.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 -- 172.21.15.38:0/1720436730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c100210 msgr2=0x7f6e0c199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:18.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c100210 0x7f6e0c199910 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00d770 tx=0x7f6dfc00dc30 comp rx=0 tx=0).stop 2024-03-26T06:35:18.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 -- 172.21.15.38:0/1720436730 shutdown_connections 2024-03-26T06:35:18.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6df80a2800 0x7f6df80a4cc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:18.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6e0c101dc0 0x7f6e0c1a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:18.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e0c101400 0x7f6e0c199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:18.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 --2- 172.21.15.38:0/1720436730 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6e0c100210 0x7f6e0c199910 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:18.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 -- 172.21.15.38:0/1720436730 >> 172.21.15.38:0/1720436730 conn(0x7f6e0c0fb780 msgr2=0x7f6e0c0fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:18.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 -- 172.21.15.38:0/1720436730 shutdown_connections 2024-03-26T06:35:18.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:18.190+0000 7f6e1229a640 1 -- 172.21.15.38:0/1720436730 wait complete. 2024-03-26T06:35:18.790 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:35:18 smithi038 systemd[1]: Starting Ceph mds.a for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:35:19.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:18 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1720436730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:19.073 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:35:18 smithi038 podman[71602]: 2024-03-26 06:35:18.701583401 +0000 UTC m=+0.017969761 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:35:19.073 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:35:18 smithi038 podman[71602]: 2024-03-26 06:35:18.827466645 +0000 UTC m=+0.143852976 container create 7fdcfc8c1d627b605e7e7905a7b381f3e02dc9a5a0f6d442bc8673c4902bcd33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-a, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-26T06:35:19.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:18 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1720436730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:19.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:18 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1720436730' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:19.333 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:35:19 smithi038 podman[71602]: 2024-03-26 06:35:19.071705387 +0000 UTC m=+0.388091719 container init 7fdcfc8c1d627b605e7e7905a7b381f3e02dc9a5a0f6d442bc8673c4902bcd33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-26T06:35:19.334 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:35:19 smithi038 podman[71602]: 2024-03-26 06:35:19.074323698 +0000 UTC m=+0.390710030 container start 7fdcfc8c1d627b605e7e7905a7b381f3e02dc9a5a0f6d442bc8673c4902bcd33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-a, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T06:35:19.334 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:35:19 smithi038 ceph-mds[71621]: -- 172.21.15.38:0/3966162791 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f1aefd7a40 con 0x55f1aefab000 2024-03-26T06:35:19.334 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:35:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-a[71617]: starting mds.a at 2024-03-26T06:35:19.334 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:35:19 smithi038 bash[71602]: 7fdcfc8c1d627b605e7e7905a7b381f3e02dc9a5a0f6d442bc8673c4902bcd33 2024-03-26T06:35:19.334 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:35:19 smithi038 systemd[1]: Started Ceph mds.a for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:35:20.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:19 smithi121 ceph-mon[36349]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:20.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:20.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:20.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:20.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:20.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:20.048 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:19 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:19 smithi038 ceph-mon[32265]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:20.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:20.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:19 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:20.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:19 smithi170 ceph-mon[36912]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:20.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:20.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:20.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:20.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-26T06:35:20.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-26T06:35:20.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:19 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:20.205 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:35:20.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.321+0000 7f0a09d53640 1 -- 172.21.15.38:0/1462325838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a04101480 msgr2=0x7f0a04101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:20.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.321+0000 7f0a09d53640 1 --2- 172.21.15.38:0/1462325838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a04101480 0x7f0a04101900 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f09f8009900 tx=0x7f09f802bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:20.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.321+0000 7f0a09d53640 1 -- 172.21.15.38:0/1462325838 shutdown_connections 2024-03-26T06:35:20.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.321+0000 7f0a09d53640 1 --2- 172.21.15.38:0/1462325838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a04101e40 0x7f0a041086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:20.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.321+0000 7f0a09d53640 1 --2- 172.21.15.38:0/1462325838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a04101480 0x7f0a04101900 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:20.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.321+0000 7f0a09d53640 1 --2- 172.21.15.38:0/1462325838 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a04100290 0x7f0a041006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:20.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.321+0000 7f0a09d53640 1 -- 172.21.15.38:0/1462325838 >> 172.21.15.38:0/1462325838 conn(0x7f0a040fb820 msgr2=0x7f0a040fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:20.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.321+0000 7f0a09d53640 1 -- 172.21.15.38:0/1462325838 shutdown_connections 2024-03-26T06:35:20.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.321+0000 7f0a09d53640 1 -- 172.21.15.38:0/1462325838 wait complete. 2024-03-26T06:35:20.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a09d53640 1 Processor -- start 2024-03-26T06:35:20.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a09d53640 1 -- start start 2024-03-26T06:35:20.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a09d53640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a04100290 0x7f0a04196520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:20.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a09d53640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a04101480 0x7f0a04196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a09d53640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a04101e40 0x7f0a0419dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a09d53640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0410beb0 con 0x7f0a04100290 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a09d53640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0410bd30 con 0x7f0a04101480 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a09d53640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0410c030 con 0x7f0a04101e40 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a037fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a04100290 0x7f0a04196520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a037fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a04100290 0x7f0a04196520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33694/0 (socket says 172.21.15.38:33694) 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a037fe640 1 -- 172.21.15.38:0/473534417 learned_addr learned my addr 172.21.15.38:0/473534417 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a02ffd640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a04101480 0x7f0a04196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a037fe640 1 -- 172.21.15.38:0/473534417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a04101e40 msgr2=0x7f0a0419dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a037fe640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a04101e40 0x7f0a0419dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a037fe640 1 -- 172.21.15.38:0/473534417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a04101480 msgr2=0x7f0a04196a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a037fe640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a04101480 0x7f0a04196a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:20.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a037fe640 1 -- 172.21.15.38:0/473534417 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0419e140 con 0x7f0a04100290 2024-03-26T06:35:20.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.322+0000 7f0a02ffd640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a04101480 0x7f0a04196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:20.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.323+0000 7f0a037fe640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a04100290 0x7f0a04196520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f09f400ead0 tx=0x7f09f400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:20.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.323+0000 7f0a00ff9640 1 -- 172.21.15.38:0/473534417 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f401c070 con 0x7f0a04100290 2024-03-26T06:35:20.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.323+0000 7f0a00ff9640 1 -- 172.21.15.38:0/473534417 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09f400bbd0 con 0x7f0a04100290 2024-03-26T06:35:20.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.323+0000 7f0a09d53640 1 -- 172.21.15.38:0/473534417 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a04069980 con 0x7f0a04100290 2024-03-26T06:35:20.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.323+0000 7f0a00ff9640 1 -- 172.21.15.38:0/473534417 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4003de0 con 0x7f0a04100290 2024-03-26T06:35:20.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.323+0000 7f0a09d53640 1 -- 172.21.15.38:0/473534417 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a04069ea0 con 0x7f0a04100290 2024-03-26T06:35:20.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.324+0000 7f0a09d53640 1 -- 172.21.15.38:0/473534417 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a04068530 con 0x7f0a04100290 2024-03-26T06:35:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.325+0000 7f0a00ff9640 1 -- 172.21.15.38:0/473534417 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f09f401f070 con 0x7f0a04100290 2024-03-26T06:35:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.326+0000 7f0a00ff9640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f09ec0a2880 0x7f09ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.326+0000 7f0a02ffd640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f09ec0a2880 0x7f09ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.326+0000 7f0a00ff9640 1 -- 172.21.15.38:0/473534417 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f09f40d0570 con 0x7f0a04100290 2024-03-26T06:35:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.326+0000 7f0a02ffd640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f09ec0a2880 0x7f09ec0a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f09f8006010 tx=0x7f09f802c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.330+0000 7f0a00ff9640 1 -- 172.21.15.38:0/473534417 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f09f4026070 con 0x7f0a04100290 2024-03-26T06:35:20.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.549+0000 7f0a09d53640 1 -- 172.21.15.38:0/473534417 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a041070c0 con 0x7f0a04100290 2024-03-26T06:35:20.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.549+0000 7f0a00ff9640 1 -- 172.21.15.38:0/473534417 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9254 (secure 0 0 0) 0x7f09f409da10 con 0x7f0a04100290 2024-03-26T06:35:20.552 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:20.552 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/357411663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":357411663},{"type":"v1","addr":"172.21.15.38:6839","nonce":357411663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/654319512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":654319512},{"type":"v1","addr":"172.21.15.38:6841","nonce":654319512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24502,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-26T06:35:20.552 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 12 2024-03-26T06:35:20.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.551+0000 7f0a09d53640 1 -- 172.21.15.38:0/473534417 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f09ec0a2880 msgr2=0x7f09ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:20.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.551+0000 7f0a09d53640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f09ec0a2880 0x7f09ec0a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f09f8006010 tx=0x7f09f802c8a0 comp rx=0 tx=0).stop 2024-03-26T06:35:20.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.551+0000 7f0a09d53640 1 -- 172.21.15.38:0/473534417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a04100290 msgr2=0x7f0a04196520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:20.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.551+0000 7f0a09d53640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a04100290 0x7f0a04196520 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f09f400ead0 tx=0x7f09f400ef90 comp rx=0 tx=0).stop 2024-03-26T06:35:20.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.552+0000 7f0a09d53640 1 -- 172.21.15.38:0/473534417 shutdown_connections 2024-03-26T06:35:20.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.552+0000 7f0a09d53640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f09ec0a2880 0x7f09ec0a4d40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:20.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.552+0000 7f0a09d53640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0a04101e40 0x7f0a0419dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:20.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.552+0000 7f0a09d53640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a04101480 0x7f0a04196a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:20.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.552+0000 7f0a09d53640 1 --2- 172.21.15.38:0/473534417 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0a04100290 0x7f0a04196520 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:20.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.552+0000 7f0a09d53640 1 -- 172.21.15.38:0/473534417 >> 172.21.15.38:0/473534417 conn(0x7f0a040fb820 msgr2=0x7f0a04104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:20.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.552+0000 7f0a09d53640 1 -- 172.21.15.38:0/473534417 shutdown_connections 2024-03-26T06:35:20.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:20.552+0000 7f0a09d53640 1 -- 172.21.15.38:0/473534417 wait complete. 2024-03-26T06:35:20.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:20 smithi038 ceph-mon[32265]: Deploying daemon mds.b on smithi121 2024-03-26T06:35:20.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:20 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.38:6840/654319512,v1:172.21.15.38:6841/654319512] up:boot 2024-03-26T06:35:20.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:20 smithi038 ceph-mon[32265]: fsmap 11 up:standby 2024-03-26T06:35:20.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:20 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-26T06:35:20.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:20 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/473534417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:21.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:20 smithi121 ceph-mon[36349]: Deploying daemon mds.b on smithi121 2024-03-26T06:35:21.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:20 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.38:6840/654319512,v1:172.21.15.38:6841/654319512] up:boot 2024-03-26T06:35:21.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:20 smithi121 ceph-mon[36349]: fsmap 11 up:standby 2024-03-26T06:35:21.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:20 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-26T06:35:21.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:20 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/473534417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:21.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:20 smithi170 ceph-mon[36912]: Deploying daemon mds.b on smithi121 2024-03-26T06:35:21.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:20 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.38:6840/654319512,v1:172.21.15.38:6841/654319512] up:boot 2024-03-26T06:35:21.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:20 smithi170 ceph-mon[36912]: fsmap 11 up:standby 2024-03-26T06:35:21.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:20 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-26T06:35:21.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:20 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/473534417' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:21.618 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:21 smithi121 systemd[1]: Starting Ceph mds.b for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:35:21.870 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:21 smithi121 ceph-mon[36349]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:21.870 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:21 smithi121 podman[66803]: 2024-03-26 06:35:21.61590534 +0000 UTC m=+0.101122730 container create 6b23d6d3808ec328a9f7b4dff43bc506ec0f8b08a9f36464ccd46428a7900192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T06:35:21.871 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:21 smithi121 podman[66803]: 2024-03-26 06:35:21.532696352 +0000 UTC m=+0.017913753 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:35:21.871 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:21 smithi121 podman[66803]: 2024-03-26 06:35:21.843345921 +0000 UTC m=+0.328563307 container init 6b23d6d3808ec328a9f7b4dff43bc506ec0f8b08a9f36464ccd46428a7900192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T06:35:21.871 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:21 smithi121 podman[66803]: 2024-03-26 06:35:21.846137485 +0000 UTC m=+0.331354890 container start 6b23d6d3808ec328a9f7b4dff43bc506ec0f8b08a9f36464ccd46428a7900192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5) 2024-03-26T06:35:21.871 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:21 smithi121 ceph-mds[66822]: -- 172.21.15.121:0/2138219773 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561277713a40 con 0x5612776e6800 2024-03-26T06:35:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:21 smithi038 ceph-mon[32265]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:22.146 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:21 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b[66818]: starting mds.b at 2024-03-26T06:35:22.146 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:21 smithi121 bash[66803]: 6b23d6d3808ec328a9f7b4dff43bc506ec0f8b08a9f36464ccd46428a7900192 2024-03-26T06:35:22.146 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:21 smithi121 systemd[1]: Started Ceph mds.b for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:35:22.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:21 smithi170 ceph-mon[36912]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:22.566 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:35:22.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.667+0000 7fb1e9e85640 1 -- 172.21.15.38:0/1274316054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1e4101d90 msgr2=0x7fb1e4104180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.667+0000 7fb1e9e85640 1 --2- 172.21.15.38:0/1274316054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1e4101d90 0x7fb1e4104180 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009900 tx=0x7fb1d402bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.667+0000 7fb1e9e85640 1 -- 172.21.15.38:0/1274316054 shutdown_connections 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.667+0000 7fb1e9e85640 1 --2- 172.21.15.38:0/1274316054 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e4106ff0 0x7fb1e4109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.667+0000 7fb1e9e85640 1 --2- 172.21.15.38:0/1274316054 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1e41046c0 0x7fb1e4106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.667+0000 7fb1e9e85640 1 --2- 172.21.15.38:0/1274316054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1e4101d90 0x7fb1e4104180 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.667+0000 7fb1e9e85640 1 -- 172.21.15.38:0/1274316054 >> 172.21.15.38:0/1274316054 conn(0x7fb1e40fb780 msgr2=0x7fb1e40fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.667+0000 7fb1e9e85640 1 -- 172.21.15.38:0/1274316054 shutdown_connections 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.667+0000 7fb1e9e85640 1 -- 172.21.15.38:0/1274316054 wait complete. 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e9e85640 1 Processor -- start 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e9e85640 1 -- start start 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e9e85640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e4101d90 0x7fb1e4197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e9e85640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1e41046c0 0x7fb1e4197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:22.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e9e85640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1e4106ff0 0x7fb1e419ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e9e85640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e410cfa0 con 0x7fb1e4106ff0 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e9e85640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e410ce20 con 0x7fb1e4101d90 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e9e85640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1e410d120 con 0x7fb1e41046c0 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1e4106ff0 0x7fb1e419ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e37fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e4101d90 0x7fb1e4197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1e4106ff0 0x7fb1e419ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33718/0 (socket says 172.21.15.38:33718) 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e3fff640 1 -- 172.21.15.38:0/631698199 learned_addr learned my addr 172.21.15.38:0/631698199 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e37fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e4101d90 0x7fb1e4197680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:40370/0 (socket says 172.21.15.38:40370) 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e2ffd640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1e41046c0 0x7fb1e4197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e3fff640 1 -- 172.21.15.38:0/631698199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1e41046c0 msgr2=0x7fb1e4197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e3fff640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1e41046c0 0x7fb1e4197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:22.673 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e3fff640 1 -- 172.21.15.38:0/631698199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e4101d90 msgr2=0x7fb1e4197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:22.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e3fff640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e4101d90 0x7fb1e4197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:22.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.668+0000 7fb1e3fff640 1 -- 172.21.15.38:0/631698199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e419f2a0 con 0x7fb1e4106ff0 2024-03-26T06:35:22.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.669+0000 7fb1e3fff640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1e4106ff0 0x7fb1e419ec30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00eb10 tx=0x7fb1dc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:22.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.669+0000 7fb1e0ff9640 1 -- 172.21.15.38:0/631698199 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc004b20 con 0x7fb1e4106ff0 2024-03-26T06:35:22.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.669+0000 7fb1e0ff9640 1 -- 172.21.15.38:0/631698199 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1dc004cc0 con 0x7fb1e4106ff0 2024-03-26T06:35:22.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.669+0000 7fb1e9e85640 1 -- 172.21.15.38:0/631698199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e419f590 con 0x7fb1e4106ff0 2024-03-26T06:35:22.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.669+0000 7fb1e0ff9640 1 -- 172.21.15.38:0/631698199 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1dc010760 con 0x7fb1e4106ff0 2024-03-26T06:35:22.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.669+0000 7fb1e9e85640 1 -- 172.21.15.38:0/631698199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e419fa90 con 0x7fb1e4106ff0 2024-03-26T06:35:22.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.670+0000 7fb1e9e85640 1 -- 172.21.15.38:0/631698199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e40fd270 con 0x7fb1e4106ff0 2024-03-26T06:35:22.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.671+0000 7fb1e0ff9640 1 -- 172.21.15.38:0/631698199 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb1dc005020 con 0x7fb1e4106ff0 2024-03-26T06:35:22.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.672+0000 7fb1e0ff9640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1cc0a2800 0x7fb1cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:22.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.672+0000 7fb1e37fe640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1cc0a2800 0x7fb1cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:22.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.672+0000 7fb1e0ff9640 1 -- 172.21.15.38:0/631698199 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb1dc0cebd0 con 0x7fb1e4106ff0 2024-03-26T06:35:22.676 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.672+0000 7fb1e37fe640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1cc0a2800 0x7fb1cc0a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb1d402c7b0 tx=0x7fb1d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:22.678 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.676+0000 7fb1e0ff9640 1 -- 172.21.15.38:0/631698199 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb1dc09c070 con 0x7fb1e4106ff0 2024-03-26T06:35:22.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.965+0000 7fb1e9e85640 1 -- 172.21.15.38:0/631698199 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1e4061ef0 con 0x7fb1e4106ff0 2024-03-26T06:35:22.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.965+0000 7fb1e0ff9640 1 -- 172.21.15.38:0/631698199 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10046 (secure 0 0 0) 0x7fb1dc09b970 con 0x7fb1e4106ff0 2024-03-26T06:35:22.968 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:22.968 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/357411663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":357411663},{"type":"v1","addr":"172.21.15.38:6839","nonce":357411663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/654319512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":654319512},{"type":"v1","addr":"172.21.15.38:6841","nonce":654319512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24460,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24472,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24488,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24502,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24539,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6839/359437185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":359437185},{"type":"v1","addr":"172.21.15.121:6839","nonce":359437185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-26T06:35:22.968 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 13 2024-03-26T06:35:22.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 -- 172.21.15.38:0/631698199 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1cc0a2800 msgr2=0x7fb1cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:22.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1cc0a2800 0x7fb1cc0a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb1d402c7b0 tx=0x7fb1d4005fd0 comp rx=0 tx=0).stop 2024-03-26T06:35:22.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 -- 172.21.15.38:0/631698199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1e4106ff0 msgr2=0x7fb1e419ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:22.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1e4106ff0 0x7fb1e419ec30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc00eb10 tx=0x7fb1dc00c2d0 comp rx=0 tx=0).stop 2024-03-26T06:35:22.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 -- 172.21.15.38:0/631698199 shutdown_connections 2024-03-26T06:35:22.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1cc0a2800 0x7fb1cc0a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:22.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb1e4106ff0 0x7fb1e419ec30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:22.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb1e41046c0 0x7fb1e4197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:22.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 --2- 172.21.15.38:0/631698199 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1e4101d90 0x7fb1e4197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:22.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 -- 172.21.15.38:0/631698199 >> 172.21.15.38:0/631698199 conn(0x7fb1e40fb780 msgr2=0x7fb1e4102870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:22.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 -- 172.21.15.38:0/631698199 shutdown_connections 2024-03-26T06:35:22.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:22.968+0000 7fb1e9e85640 1 -- 172.21.15.38:0/631698199 wait complete. 2024-03-26T06:35:22.981 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:35:23.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.083+0000 7f4172232640 1 -- 172.21.15.38:0/3997240336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f416c107010 msgr2=0x7f416c109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.083+0000 7f4172232640 1 --2- 172.21.15.38:0/3997240336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f416c107010 0x7f416c109460 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4164009ca0 tx=0x7f416402c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.083+0000 7f4172232640 1 -- 172.21.15.38:0/3997240336 shutdown_connections 2024-03-26T06:35:23.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.083+0000 7f4172232640 1 --2- 172.21.15.38:0/3997240336 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f416c107010 0x7f416c109460 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.083+0000 7f4172232640 1 --2- 172.21.15.38:0/3997240336 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c1046e0 0x7f416c106ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.083+0000 7f4172232640 1 --2- 172.21.15.38:0/3997240336 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f416c101db0 0x7f416c1041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.083+0000 7f4172232640 1 -- 172.21.15.38:0/3997240336 >> 172.21.15.38:0/3997240336 conn(0x7f416c0fb7a0 msgr2=0x7f416c0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:23.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.083+0000 7f4172232640 1 -- 172.21.15.38:0/3997240336 shutdown_connections 2024-03-26T06:35:23.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.083+0000 7f4172232640 1 -- 172.21.15.38:0/3997240336 wait complete. 2024-03-26T06:35:23.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f4172232640 1 Processor -- start 2024-03-26T06:35:23.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f4172232640 1 -- start start 2024-03-26T06:35:23.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f4172232640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f416c101db0 0x7f416c197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f4172232640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f416c1046e0 0x7f416c197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f4172232640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c107010 0x7f416c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f4172232640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c10cfc0 con 0x7f416c1046e0 2024-03-26T06:35:23.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f4172232640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c10ce40 con 0x7f416c107010 2024-03-26T06:35:23.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f4172232640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f416c10d140 con 0x7f416c101db0 2024-03-26T06:35:23.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f416affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f416c1046e0 0x7f416c197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f416b7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f416c101db0 0x7f416c197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f416affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f416c1046e0 0x7f416c197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33726/0 (socket says 172.21.15.38:33726) 2024-03-26T06:35:23.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f416affd640 1 -- 172.21.15.38:0/4192193286 learned_addr learned my addr 172.21.15.38:0/4192193286 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:23.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f416bfff640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c107010 0x7f416c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f416affd640 1 -- 172.21.15.38:0/4192193286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f416c101db0 msgr2=0x7f416c197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f416affd640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f416c101db0 0x7f416c197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f416affd640 1 -- 172.21.15.38:0/4192193286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c107010 msgr2=0x7f416c19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f416affd640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c107010 0x7f416c19eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.084+0000 7f416affd640 1 -- 172.21.15.38:0/4192193286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c19f1f0 con 0x7f416c1046e0 2024-03-26T06:35:23.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.085+0000 7f416affd640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f416c1046e0 0x7f416c197b60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f416000e9c0 tx=0x7f416000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:23.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.085+0000 7f4168ff9640 1 -- 172.21.15.38:0/4192193286 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416000cd80 con 0x7f416c1046e0 2024-03-26T06:35:23.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.085+0000 7f4168ff9640 1 -- 172.21.15.38:0/4192193286 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4160005500 con 0x7f416c1046e0 2024-03-26T06:35:23.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.085+0000 7f4172232640 1 -- 172.21.15.38:0/4192193286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416c19f4e0 con 0x7f416c1046e0 2024-03-26T06:35:23.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.085+0000 7f4168ff9640 1 -- 172.21.15.38:0/4192193286 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4160010690 con 0x7f416c1046e0 2024-03-26T06:35:23.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.085+0000 7f4172232640 1 -- 172.21.15.38:0/4192193286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416c19fa30 con 0x7f416c1046e0 2024-03-26T06:35:23.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.086+0000 7f4172232640 1 -- 172.21.15.38:0/4192193286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416c0fd290 con 0x7f416c1046e0 2024-03-26T06:35:23.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.087+0000 7f4168ff9640 1 -- 172.21.15.38:0/4192193286 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f41600046c0 con 0x7f416c1046e0 2024-03-26T06:35:23.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.087+0000 7f4168ff9640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f41540a2800 0x7f41540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.088+0000 7f416b7fe640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f41540a2800 0x7f41540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.088+0000 7f4168ff9640 1 -- 172.21.15.38:0/4192193286 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f41600ceb30 con 0x7f416c1046e0 2024-03-26T06:35:23.091 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.088+0000 7f416b7fe640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f41540a2800 0x7f41540a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f415c009bd0 tx=0x7f415c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:23.093 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.091+0000 7f4168ff9640 1 -- 172.21.15.38:0/4192193286 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4160053260 con 0x7f416c1046e0 2024-03-26T06:35:23.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:35:23.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:23 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] up:boot 2024-03-26T06:35:23.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:23 smithi038 ceph-mon[32265]: fsmap 12 up:standby 2024-03-26T06:35:23.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:23 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-26T06:35:23.260 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:23 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/631698199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:23.282 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.280+0000 7f4172232640 1 -- 172.21.15.38:0/4192193286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f416c02d0c0 con 0x7f416c1046e0 2024-03-26T06:35:23.283 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.281+0000 7f4168ff9640 1 -- 172.21.15.38:0/4192193286 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18446 (secure 0 0 0) 0x7f416009c070 con 0x7f416c1046e0 2024-03-26T06:35:23.283 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:23.283 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":82,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:34:37.602828+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:35:23.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 -- 172.21.15.38:0/4192193286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f41540a2800 msgr2=0x7f41540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f41540a2800 0x7f41540a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f415c009bd0 tx=0x7f415c005f90 comp rx=0 tx=0).stop 2024-03-26T06:35:23.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 -- 172.21.15.38:0/4192193286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f416c1046e0 msgr2=0x7f416c197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f416c1046e0 0x7f416c197b60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f416000e9c0 tx=0x7f416000ee80 comp rx=0 tx=0).stop 2024-03-26T06:35:23.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 -- 172.21.15.38:0/4192193286 shutdown_connections 2024-03-26T06:35:23.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f41540a2800 0x7f41540a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f416c107010 0x7f416c19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f416c1046e0 0x7f416c197b60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 --2- 172.21.15.38:0/4192193286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f416c101db0 0x7f416c197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 -- 172.21.15.38:0/4192193286 >> 172.21.15.38:0/4192193286 conn(0x7f416c0fb7a0 msgr2=0x7f416c1026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:23.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 -- 172.21.15.38:0/4192193286 shutdown_connections 2024-03-26T06:35:23.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.283+0000 7f4172232640 1 -- 172.21.15.38:0/4192193286 wait complete. 2024-03-26T06:35:23.296 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T06:30:52.485897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T06:35:23.296 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T06:35:23.299 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.300 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.300 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.300 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.300 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.300 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:35:23.300 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:23 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] up:boot 2024-03-26T06:35:23.300 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:23 smithi170 ceph-mon[36912]: fsmap 12 up:standby 2024-03-26T06:35:23.300 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:23 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-26T06:35:23.300 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:23 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/631698199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:23.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:23.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:35:23.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:23 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] up:boot 2024-03-26T06:35:23.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:23 smithi121 ceph-mon[36349]: fsmap 12 up:standby 2024-03-26T06:35:23.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:23 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-26T06:35:23.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:23 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/631698199' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:23.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.400+0000 7fbc79f59640 1 -- 172.21.15.38:0/2010588684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc741001f0 msgr2=0x7fbc74100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.400+0000 7fbc79f59640 1 --2- 172.21.15.38:0/2010588684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc741001f0 0x7fbc74100610 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009900 tx=0x7fbc6402bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:23.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 -- 172.21.15.38:0/2010588684 shutdown_connections 2024-03-26T06:35:23.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 --2- 172.21.15.38:0/2010588684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc74101da0 0x7fbc74108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 --2- 172.21.15.38:0/2010588684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc741013e0 0x7fbc74101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.405 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 --2- 172.21.15.38:0/2010588684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc741001f0 0x7fbc74100610 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 -- 172.21.15.38:0/2010588684 >> 172.21.15.38:0/2010588684 conn(0x7fbc740fb780 msgr2=0x7fbc740fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 -- 172.21.15.38:0/2010588684 shutdown_connections 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 -- 172.21.15.38:0/2010588684 wait complete. 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 Processor -- start 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 -- start start 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc741001f0 0x7fbc74068fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc741013e0 0x7fbc7419c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc74101da0 0x7fbc7419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7410be80 con 0x7fbc741001f0 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7410bd00 con 0x7fbc74101da0 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.401+0000 7fbc79f59640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7410c000 con 0x7fbc741013e0 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc741001f0 0x7fbc74068fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.406 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc737fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc741001f0 0x7fbc74068fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33748/0 (socket says 172.21.15.38:33748) 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc737fe640 1 -- 172.21.15.38:0/1759765418 learned_addr learned my addr 172.21.15.38:0/1759765418 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc72ffd640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc741013e0 0x7fbc7419c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc73fff640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc74101da0 0x7fbc7419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc737fe640 1 -- 172.21.15.38:0/1759765418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc741013e0 msgr2=0x7fbc7419c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc737fe640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc741013e0 0x7fbc7419c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc737fe640 1 -- 172.21.15.38:0/1759765418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc74101da0 msgr2=0x7fbc7419e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc737fe640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc74101da0 0x7fbc7419e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc737fe640 1 -- 172.21.15.38:0/1759765418 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7419f290 con 0x7fbc741001f0 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc737fe640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc741001f0 0x7fbc74068fe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbc64000c00 tx=0x7fbc64005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc70ff9640 1 -- 172.21.15.38:0/1759765418 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64041070 con 0x7fbc741001f0 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc70ff9640 1 -- 172.21.15.38:0/1759765418 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc640044d0 con 0x7fbc741001f0 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc79f59640 1 -- 172.21.15.38:0/1759765418 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc7419f520 con 0x7fbc741001f0 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc70ff9640 1 -- 172.21.15.38:0/1759765418 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64034440 con 0x7fbc741001f0 2024-03-26T06:35:23.407 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.402+0000 7fbc79f59640 1 -- 172.21.15.38:0/1759765418 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7419fa10 con 0x7fbc741001f0 2024-03-26T06:35:23.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.403+0000 7fbc79f59640 1 -- 172.21.15.38:0/1759765418 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc741075b0 con 0x7fbc741001f0 2024-03-26T06:35:23.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.404+0000 7fbc70ff9640 1 -- 172.21.15.38:0/1759765418 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbc64044070 con 0x7fbc741001f0 2024-03-26T06:35:23.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.405+0000 7fbc70ff9640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbc600a2800 0x7fbc600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.405+0000 7fbc70ff9640 1 -- 172.21.15.38:0/1759765418 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbc640f02d0 con 0x7fbc741001f0 2024-03-26T06:35:23.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.405+0000 7fbc72ffd640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbc600a2800 0x7fbc600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.405+0000 7fbc72ffd640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbc600a2800 0x7fbc600a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbc74068d50 tx=0x7fbc5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:23.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.409+0000 7fbc70ff9640 1 -- 172.21.15.38:0/1759765418 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbc640bd1c0 con 0x7fbc741001f0 2024-03-26T06:35:23.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.596+0000 7fbc79f59640 1 -- 172.21.15.38:0/1759765418 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbc7402d000 con 0x7fbc741001f0 2024-03-26T06:35:23.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.596+0000 7fbc70ff9640 1 -- 172.21.15.38:0/1759765418 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbc640c1260 con 0x7fbc741001f0 2024-03-26T06:35:23.599 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 -- 172.21.15.38:0/1759765418 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbc600a2800 msgr2=0x7fbc600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbc600a2800 0x7fbc600a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fbc74068d50 tx=0x7fbc5c005f90 comp rx=0 tx=0).stop 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 -- 172.21.15.38:0/1759765418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc741001f0 msgr2=0x7fbc74068fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc741001f0 0x7fbc74068fe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbc64000c00 tx=0x7fbc64005d50 comp rx=0 tx=0).stop 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 -- 172.21.15.38:0/1759765418 shutdown_connections 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbc600a2800 0x7fbc600a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc74101da0 0x7fbc7419e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbc741013e0 0x7fbc7419c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 --2- 172.21.15.38:0/1759765418 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc741001f0 0x7fbc74068fe0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 -- 172.21.15.38:0/1759765418 >> 172.21.15.38:0/1759765418 conn(0x7fbc740fb780 msgr2=0x7fbc74104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 -- 172.21.15.38:0/1759765418 shutdown_connections 2024-03-26T06:35:23.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.598+0000 7fbc79f59640 1 -- 172.21.15.38:0/1759765418 wait complete. 2024-03-26T06:35:23.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-26T06:35:23.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.724+0000 7f5a13b90640 1 -- 172.21.15.38:0/875429235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a0c101dd0 msgr2=0x7f5a0c108660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.724+0000 7f5a13b90640 1 --2- 172.21.15.38:0/875429235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a0c101dd0 0x7f5a0c108660 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009ca0 tx=0x7f5a0402c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.724+0000 7f5a13b90640 1 -- 172.21.15.38:0/875429235 shutdown_connections 2024-03-26T06:35:23.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.724+0000 7f5a13b90640 1 --2- 172.21.15.38:0/875429235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a0c101dd0 0x7f5a0c108660 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.724+0000 7f5a13b90640 1 --2- 172.21.15.38:0/875429235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0c101410 0x7f5a0c101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.724+0000 7f5a13b90640 1 --2- 172.21.15.38:0/875429235 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a0c100220 0x7f5a0c100640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.724+0000 7f5a13b90640 1 -- 172.21.15.38:0/875429235 >> 172.21.15.38:0/875429235 conn(0x7f5a0c0fb7d0 msgr2=0x7f5a0c0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:23.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.724+0000 7f5a13b90640 1 -- 172.21.15.38:0/875429235 shutdown_connections 2024-03-26T06:35:23.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.724+0000 7f5a13b90640 1 -- 172.21.15.38:0/875429235 wait complete. 2024-03-26T06:35:23.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a13b90640 1 Processor -- start 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a13b90640 1 -- start start 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a13b90640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0c100220 0x7f5a0c196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a13b90640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a0c101410 0x7f5a0c196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a13b90640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a0c101dd0 0x7f5a0c19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a13b90640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c10beb0 con 0x7f5a0c101410 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a13b90640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c10bd30 con 0x7f5a0c100220 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a13b90640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c10c030 con 0x7f5a0c101dd0 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a1112b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a0c101410 0x7f5a0c196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a1112b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a0c101410 0x7f5a0c196a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33774/0 (socket says 172.21.15.38:33774) 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a1112b640 1 -- 172.21.15.38:0/2339232721 learned_addr learned my addr 172.21.15.38:0/2339232721 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a1192c640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0c100220 0x7f5a0c196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.725+0000 7f5a1212d640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a0c101dd0 0x7f5a0c19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.726+0000 7f5a1112b640 1 -- 172.21.15.38:0/2339232721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a0c101dd0 msgr2=0x7f5a0c19daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.726+0000 7f5a1112b640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a0c101dd0 0x7f5a0c19daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.726+0000 7f5a1112b640 1 -- 172.21.15.38:0/2339232721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0c100220 msgr2=0x7f5a0c196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.726+0000 7f5a1112b640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0c100220 0x7f5a0c196540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.726+0000 7f5a1112b640 1 -- 172.21.15.38:0/2339232721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c19e160 con 0x7f5a0c101410 2024-03-26T06:35:23.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.726+0000 7f5a1112b640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a0c101410 0x7f5a0c196a80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00e9f0 tx=0x7f59fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:23.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.726+0000 7f5a02ffd640 1 -- 172.21.15.38:0/2339232721 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc00cd80 con 0x7f5a0c101410 2024-03-26T06:35:23.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.726+0000 7f5a13b90640 1 -- 172.21.15.38:0/2339232721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0c19e450 con 0x7f5a0c101410 2024-03-26T06:35:23.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.726+0000 7f5a02ffd640 1 -- 172.21.15.38:0/2339232721 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59fc010910 con 0x7f5a0c101410 2024-03-26T06:35:23.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.726+0000 7f5a02ffd640 1 -- 172.21.15.38:0/2339232721 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc018990 con 0x7f5a0c101410 2024-03-26T06:35:23.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.726+0000 7f5a13b90640 1 -- 172.21.15.38:0/2339232721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0c19e9a0 con 0x7f5a0c101410 2024-03-26T06:35:23.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.727+0000 7f5a13b90640 1 -- 172.21.15.38:0/2339232721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0c1074e0 con 0x7f5a0c101410 2024-03-26T06:35:23.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.728+0000 7f5a02ffd640 1 -- 172.21.15.38:0/2339232721 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f59fc010ac0 con 0x7f5a0c101410 2024-03-26T06:35:23.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.729+0000 7f5a02ffd640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f59f80a2800 0x7f59f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:23.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.729+0000 7f5a1192c640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f59f80a2800 0x7f59f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:23.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.729+0000 7f5a02ffd640 1 -- 172.21.15.38:0/2339232721 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f59fc0ced90 con 0x7f5a0c101410 2024-03-26T06:35:23.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.729+0000 7f5a1192c640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f59f80a2800 0x7f59f80a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c101270 tx=0x7f5a08005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:23.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.734+0000 7f5a02ffd640 1 -- 172.21.15.38:0/2339232721 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f59fc09c2d0 con 0x7f5a0c101410 2024-03-26T06:35:23.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.935+0000 7f5a13b90640 1 -- 172.21.15.38:0/2339232721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5a0c02d0e0 con 0x7f5a0c101410 2024-03-26T06:35:23.937 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.935+0000 7f5a02ffd640 1 -- 172.21.15.38:0/2339232721 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7f59fc09bbd0 con 0x7f5a0c101410 2024-03-26T06:35:23.938 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:23.938 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2024-03-26T06:35:23.938 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:35:23.938 INFO:teuthology.orchestra.run.smithi038.stdout: "poolnum": 1, 2024-03-26T06:35:23.938 INFO:teuthology.orchestra.run.smithi038.stdout: "poolname": ".mgr" 2024-03-26T06:35:23.938 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:35:23.938 INFO:teuthology.orchestra.run.smithi038.stdout:] 2024-03-26T06:35:23.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 -- 172.21.15.38:0/2339232721 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f59f80a2800 msgr2=0x7f59f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f59f80a2800 0x7f59f80a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c101270 tx=0x7f5a08005f90 comp rx=0 tx=0).stop 2024-03-26T06:35:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 -- 172.21.15.38:0/2339232721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a0c101410 msgr2=0x7f5a0c196a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a0c101410 0x7f5a0c196a80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00e9f0 tx=0x7f59fc00eeb0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 -- 172.21.15.38:0/2339232721 shutdown_connections 2024-03-26T06:35:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f59f80a2800 0x7f59f80a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5a0c101dd0 0x7f5a0c19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a0c101410 0x7f5a0c196a80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 --2- 172.21.15.38:0/2339232721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5a0c100220 0x7f5a0c196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 -- 172.21.15.38:0/2339232721 >> 172.21.15.38:0/2339232721 conn(0x7f5a0c0fb7d0 msgr2=0x7f5a0c1046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 -- 172.21.15.38:0/2339232721 shutdown_connections 2024-03-26T06:35:23.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:23.938+0000 7f5a13b90640 1 -- 172.21.15.38:0/2339232721 wait complete. 2024-03-26T06:35:23.955 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-26T06:35:23.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-26T06:35:24.070 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.063+0000 7ff8ee129640 1 -- 172.21.15.38:0/2870797052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8e8104760 msgr2=0x7ff8e8106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:24.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.063+0000 7ff8ee129640 1 --2- 172.21.15.38:0/2870797052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8e8104760 0x7ff8e8106b50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009900 tx=0x7ff8dc02bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:24.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.063+0000 7ff8ee129640 1 -- 172.21.15.38:0/2870797052 shutdown_connections 2024-03-26T06:35:24.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.063+0000 7ff8ee129640 1 --2- 172.21.15.38:0/2870797052 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8e8107090 0x7ff8e8109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:24.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.063+0000 7ff8ee129640 1 --2- 172.21.15.38:0/2870797052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8e8104760 0x7ff8e8106b50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:24.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.063+0000 7ff8ee129640 1 --2- 172.21.15.38:0/2870797052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8e8101e30 0x7ff8e8104220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:24.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.063+0000 7ff8ee129640 1 -- 172.21.15.38:0/2870797052 >> 172.21.15.38:0/2870797052 conn(0x7ff8e80fb780 msgr2=0x7ff8e80fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:24.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.063+0000 7ff8ee129640 1 -- 172.21.15.38:0/2870797052 shutdown_connections 2024-03-26T06:35:24.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.063+0000 7ff8ee129640 1 -- 172.21.15.38:0/2870797052 wait complete. 2024-03-26T06:35:24.071 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8ee129640 1 Processor -- start 2024-03-26T06:35:24.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8ee129640 1 -- start start 2024-03-26T06:35:24.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8ee129640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8e8101e30 0x7ff8e8197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:24.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8ee129640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8e8104760 0x7ff8e8197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:24.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8ee129640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8e8107090 0x7ff8e819ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:24.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8ee129640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e810cfe0 con 0x7ff8e8101e30 2024-03-26T06:35:24.072 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8ee129640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e810ce60 con 0x7ff8e8104760 2024-03-26T06:35:24.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8ee129640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e810d160 con 0x7ff8e8107090 2024-03-26T06:35:24.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8e77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8e8101e30 0x7ff8e8197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:24.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8e77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8e8101e30 0x7ff8e8197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:33794/0 (socket says 172.21.15.38:33794) 2024-03-26T06:35:24.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8e77fe640 1 -- 172.21.15.38:0/3982587992 learned_addr learned my addr 172.21.15.38:0/3982587992 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:24.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8e7fff640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8e8107090 0x7ff8e819ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:24.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.064+0000 7ff8e6ffd640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8e8104760 0x7ff8e8197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:24.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.065+0000 7ff8e77fe640 1 -- 172.21.15.38:0/3982587992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8e8107090 msgr2=0x7ff8e819ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:24.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.065+0000 7ff8e77fe640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8e8107090 0x7ff8e819ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:24.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.065+0000 7ff8e77fe640 1 -- 172.21.15.38:0/3982587992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8e8104760 msgr2=0x7ff8e8197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:24.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.065+0000 7ff8e77fe640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8e8104760 0x7ff8e8197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:24.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.065+0000 7ff8e77fe640 1 -- 172.21.15.38:0/3982587992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e819f2b0 con 0x7ff8e8101e30 2024-03-26T06:35:24.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.065+0000 7ff8e77fe640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8e8101e30 0x7ff8e8197690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800d950 tx=0x7ff8d800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:24.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.065+0000 7ff8e4ff9640 1 -- 172.21.15.38:0/3982587992 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d8017030 con 0x7ff8e8101e30 2024-03-26T06:35:24.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.065+0000 7ff8e4ff9640 1 -- 172.21.15.38:0/3982587992 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8d8004020 con 0x7ff8e8101e30 2024-03-26T06:35:24.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.065+0000 7ff8ee129640 1 -- 172.21.15.38:0/3982587992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e819f5a0 con 0x7ff8e8101e30 2024-03-26T06:35:24.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.065+0000 7ff8e4ff9640 1 -- 172.21.15.38:0/3982587992 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8d80047a0 con 0x7ff8e8101e30 2024-03-26T06:35:24.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.065+0000 7ff8ee129640 1 -- 172.21.15.38:0/3982587992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e819fb10 con 0x7ff8e8101e30 2024-03-26T06:35:24.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.066+0000 7ff8ee129640 1 -- 172.21.15.38:0/3982587992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e80fd270 con 0x7ff8e8101e30 2024-03-26T06:35:24.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.067+0000 7ff8e4ff9640 1 -- 172.21.15.38:0/3982587992 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff8d80041d0 con 0x7ff8e8101e30 2024-03-26T06:35:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.068+0000 7ff8e4ff9640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8d00a2880 0x7ff8d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.068+0000 7ff8e6ffd640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8d00a2880 0x7ff8d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.068+0000 7ff8e4ff9640 1 -- 172.21.15.38:0/3982587992 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff8d80cdfc0 con 0x7ff8e8101e30 2024-03-26T06:35:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.068+0000 7ff8e6ffd640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8d00a2880 0x7ff8d00a4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc02c170 tx=0x7ff8dc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:24.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.072+0000 7ff8e4ff9640 1 -- 172.21.15.38:0/3982587992 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff8d809b460 con 0x7ff8e8101e30 2024-03-26T06:35:24.256 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:24.253+0000 7ff8ee129640 1 -- 172.21.15.38:0/3982587992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ff8e802ce70 con 0x7ff8e8101e30 2024-03-26T06:35:24.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:24 smithi038 ceph-mon[32265]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:24.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:24 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/4192193286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:24.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:24 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1759765418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:35:24.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:24 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:24 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:24 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2339232721' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-26T06:35:24.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:24 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:24 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:24 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:24 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:24 smithi121 ceph-mon[36349]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:24 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/4192193286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:24 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1759765418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:35:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:24 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:24 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:24 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2339232721' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-26T06:35:24.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:24 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:24 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:24 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:24 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:24 smithi170 ceph-mon[36912]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:24.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:24 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/4192193286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:24.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:24 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1759765418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:35:24.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:24 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:24 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:24 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2339232721' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-26T06:35:24.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:24 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:24 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:24 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:24.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:24 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:25.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.081+0000 7ff8e4ff9640 1 -- 172.21.15.38:0/3982587992 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff8d809ad60 con 0x7ff8e8101e30 2024-03-26T06:35:25.083 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_metadata' created 2024-03-26T06:35:25.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.083+0000 7ff8ee129640 1 -- 172.21.15.38:0/3982587992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8d00a2880 msgr2=0x7ff8d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:25.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.083+0000 7ff8ee129640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8d00a2880 0x7ff8d00a4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc02c170 tx=0x7ff8dc02df90 comp rx=0 tx=0).stop 2024-03-26T06:35:25.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.083+0000 7ff8ee129640 1 -- 172.21.15.38:0/3982587992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8e8101e30 msgr2=0x7ff8e8197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:25.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.083+0000 7ff8ee129640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8e8101e30 0x7ff8e8197690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff8d800d950 tx=0x7ff8d800de10 comp rx=0 tx=0).stop 2024-03-26T06:35:25.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.083+0000 7ff8ee129640 1 -- 172.21.15.38:0/3982587992 shutdown_connections 2024-03-26T06:35:25.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.083+0000 7ff8ee129640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8d00a2880 0x7ff8d00a4d40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:25.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.083+0000 7ff8ee129640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff8e8107090 0x7ff8e819ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:25.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.083+0000 7ff8ee129640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8e8104760 0x7ff8e8197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:25.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.083+0000 7ff8ee129640 1 --2- 172.21.15.38:0/3982587992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8e8101e30 0x7ff8e8197690 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:25.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.083+0000 7ff8ee129640 1 -- 172.21.15.38:0/3982587992 >> 172.21.15.38:0/3982587992 conn(0x7ff8e80fb780 msgr2=0x7ff8e81026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:25.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.084+0000 7ff8ee129640 1 -- 172.21.15.38:0/3982587992 shutdown_connections 2024-03-26T06:35:25.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.084+0000 7ff8ee129640 1 -- 172.21.15.38:0/3982587992 wait complete. 2024-03-26T06:35:25.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-26T06:35:25.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.221+0000 7f8747d45640 1 -- 172.21.15.38:0/1441726895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8740107010 msgr2=0x7f8740109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:25.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.221+0000 7f8747d45640 1 --2- 172.21.15.38:0/1441726895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8740107010 0x7f8740109460 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f8738009ca0 tx=0x7f873802c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:25.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.221+0000 7f8747d45640 1 -- 172.21.15.38:0/1441726895 shutdown_connections 2024-03-26T06:35:25.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.221+0000 7f8747d45640 1 --2- 172.21.15.38:0/1441726895 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8740107010 0x7f8740109460 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:25.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.221+0000 7f8747d45640 1 --2- 172.21.15.38:0/1441726895 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f87401046e0 0x7f8740106ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.221+0000 7f8747d45640 1 --2- 172.21.15.38:0/1441726895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8740101db0 0x7f87401041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.221+0000 7f8747d45640 1 -- 172.21.15.38:0/1441726895 >> 172.21.15.38:0/1441726895 conn(0x7f87400fb7c0 msgr2=0x7f87400fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.221+0000 7f8747d45640 1 -- 172.21.15.38:0/1441726895 shutdown_connections 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.221+0000 7f8747d45640 1 -- 172.21.15.38:0/1441726895 wait complete. 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.221+0000 7f8747d45640 1 Processor -- start 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.221+0000 7f8747d45640 1 -- start start 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f8747d45640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8740101db0 0x7f8740101720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f8747d45640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87401046e0 0x7f87400ffd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f8747d45640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8740107010 0x7f87401002a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f8747d45640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874010cfc0 con 0x7f87401046e0 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f8747d45640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874010ce40 con 0x7f8740101db0 2024-03-26T06:35:25.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f8747d45640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f874010d140 con 0x7f8740107010 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f87452e0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87401046e0 0x7f87400ffd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f87452e0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87401046e0 0x7f87400ffd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55664/0 (socket says 172.21.15.38:55664) 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f87452e0640 1 -- 172.21.15.38:0/982831581 learned_addr learned my addr 172.21.15.38:0/982831581 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f8745ae1640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8740101db0 0x7f8740101720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f87462e2640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8740107010 0x7f87401002a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f87452e0640 1 -- 172.21.15.38:0/982831581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8740107010 msgr2=0x7f87401002a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f87452e0640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8740107010 0x7f87401002a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f87452e0640 1 -- 172.21.15.38:0/982831581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8740101db0 msgr2=0x7f8740101720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f87452e0640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8740101db0 0x7f8740101720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f87452e0640 1 -- 172.21.15.38:0/982831581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8740100b30 con 0x7f87401046e0 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f87452e0640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87401046e0 0x7f87400ffd60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f873000e9c0 tx=0x7f873000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f8736ffd640 1 -- 172.21.15.38:0/982831581 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873000cd80 con 0x7f87401046e0 2024-03-26T06:35:25.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f8747d45640 1 -- 172.21.15.38:0/982831581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87401a16b0 con 0x7f87401046e0 2024-03-26T06:35:25.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f8736ffd640 1 -- 172.21.15.38:0/982831581 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8730005500 con 0x7f87401046e0 2024-03-26T06:35:25.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.222+0000 7f8736ffd640 1 -- 172.21.15.38:0/982831581 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8730010690 con 0x7f87401046e0 2024-03-26T06:35:25.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.223+0000 7f8747d45640 1 -- 172.21.15.38:0/982831581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87401a1c00 con 0x7f87401046e0 2024-03-26T06:35:25.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.223+0000 7f8747d45640 1 -- 172.21.15.38:0/982831581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87400fd2b0 con 0x7f87401046e0 2024-03-26T06:35:25.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.225+0000 7f8736ffd640 1 -- 172.21.15.38:0/982831581 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f87300046c0 con 0x7f87401046e0 2024-03-26T06:35:25.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.225+0000 7f8736ffd640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f872c0a2800 0x7f872c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:25.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.225+0000 7f8745ae1640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f872c0a2800 0x7f872c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:25.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.225+0000 7f8736ffd640 1 -- 172.21.15.38:0/982831581 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f87300cece0 con 0x7f87401046e0 2024-03-26T06:35:25.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.226+0000 7f8745ae1640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f872c0a2800 0x7f872c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f873c009bd0 tx=0x7f873c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:25.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.233+0000 7f8736ffd640 1 -- 172.21.15.38:0/982831581 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8730053260 con 0x7f87401046e0 2024-03-26T06:35:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:25 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3982587992' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-26T06:35:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:35:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:25.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:25 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:35:25.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:25 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3982587992' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-26T06:35:25.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:25 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:25.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:25 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:35:25.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:25 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:25.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:25 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:35:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:25 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3982587992' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-26T06:35:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:35:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:25.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:25 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:35:25.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:25.427+0000 7f8747d45640 1 -- 172.21.15.38:0/982831581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f874002ce70 con 0x7f87401046e0 2024-03-26T06:35:26.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.083+0000 7f8736ffd640 1 -- 172.21.15.38:0/982831581 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) v1 ==== 170+0+0 (secure 0 0 0) 0x7f873009c0b0 con 0x7f87401046e0 2024-03-26T06:35:26.086 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data' created 2024-03-26T06:35:26.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.085+0000 7f8747d45640 1 -- 172.21.15.38:0/982831581 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f872c0a2800 msgr2=0x7f872c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:26.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.085+0000 7f8747d45640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f872c0a2800 0x7f872c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f873c009bd0 tx=0x7f873c005f90 comp rx=0 tx=0).stop 2024-03-26T06:35:26.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.085+0000 7f8747d45640 1 -- 172.21.15.38:0/982831581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87401046e0 msgr2=0x7f87400ffd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:26.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.085+0000 7f8747d45640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87401046e0 0x7f87400ffd60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f873000e9c0 tx=0x7f873000ee80 comp rx=0 tx=0).stop 2024-03-26T06:35:26.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.086+0000 7f8747d45640 1 -- 172.21.15.38:0/982831581 shutdown_connections 2024-03-26T06:35:26.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.086+0000 7f8747d45640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f872c0a2800 0x7f872c0a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:26.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.086+0000 7f8747d45640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8740107010 0x7f87401002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:26.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.086+0000 7f8747d45640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87401046e0 0x7f87400ffd60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:26.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.086+0000 7f8747d45640 1 --2- 172.21.15.38:0/982831581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8740101db0 0x7f8740101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:26.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.086+0000 7f8747d45640 1 -- 172.21.15.38:0/982831581 >> 172.21.15.38:0/982831581 conn(0x7f87400fb7c0 msgr2=0x7f87401026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:26.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.088+0000 7f8747d45640 1 -- 172.21.15.38:0/982831581 shutdown_connections 2024-03-26T06:35:26.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.092+0000 7f8747d45640 1 -- 172.21.15.38:0/982831581 wait complete. 2024-03-26T06:35:26.122 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-26T06:35:26.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.233+0000 7fc141cc0640 1 -- 172.21.15.38:0/23760912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc13c1013e0 msgr2=0x7fc13c101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.233+0000 7fc141cc0640 1 --2- 172.21.15.38:0/23760912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc13c1013e0 0x7fc13c101860 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc130009900 tx=0x7fc13002bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.233+0000 7fc141cc0640 1 -- 172.21.15.38:0/23760912 shutdown_connections 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.233+0000 7fc141cc0640 1 --2- 172.21.15.38:0/23760912 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc13c101da0 0x7fc13c108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.233+0000 7fc141cc0640 1 --2- 172.21.15.38:0/23760912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc13c1013e0 0x7fc13c101860 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.233+0000 7fc141cc0640 1 --2- 172.21.15.38:0/23760912 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c1001f0 0x7fc13c100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.233+0000 7fc141cc0640 1 -- 172.21.15.38:0/23760912 >> 172.21.15.38:0/23760912 conn(0x7fc13c0fb780 msgr2=0x7fc13c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.234+0000 7fc141cc0640 1 -- 172.21.15.38:0/23760912 shutdown_connections 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.234+0000 7fc141cc0640 1 -- 172.21.15.38:0/23760912 wait complete. 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.234+0000 7fc141cc0640 1 Processor -- start 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.234+0000 7fc141cc0640 1 -- start start 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.234+0000 7fc141cc0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc13c1001f0 0x7fc13c068fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:26.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.234+0000 7fc141cc0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c1013e0 0x7fc13c19c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.234+0000 7fc141cc0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc13c101da0 0x7fc13c19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.234+0000 7fc141cc0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c10be80 con 0x7fc13c1001f0 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.234+0000 7fc141cc0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c10bd00 con 0x7fc13c1013e0 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.234+0000 7fc141cc0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c10c000 con 0x7fc13c101da0 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.235+0000 7fc13b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc13c1001f0 0x7fc13c068fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.235+0000 7fc13b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc13c1001f0 0x7fc13c068fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55682/0 (socket says 172.21.15.38:55682) 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.235+0000 7fc13b7fe640 1 -- 172.21.15.38:0/24383843 learned_addr learned my addr 172.21.15.38:0/24383843 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.235+0000 7fc13bfff640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc13c101da0 0x7fc13c19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.235+0000 7fc13affd640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c1013e0 0x7fc13c19c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.235+0000 7fc13b7fe640 1 -- 172.21.15.38:0/24383843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc13c101da0 msgr2=0x7fc13c19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.235+0000 7fc13b7fe640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc13c101da0 0x7fc13c19e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.235+0000 7fc13b7fe640 1 -- 172.21.15.38:0/24383843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c1013e0 msgr2=0x7fc13c19c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.235+0000 7fc13b7fe640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c1013e0 0x7fc13c19c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:26.239 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.235+0000 7fc13b7fe640 1 -- 172.21.15.38:0/24383843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c19f2e0 con 0x7fc13c1001f0 2024-03-26T06:35:26.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.235+0000 7fc13b7fe640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc13c1001f0 0x7fc13c068fc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e9c0 tx=0x7fc12c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:26.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.237+0000 7fc138ff9640 1 -- 172.21.15.38:0/24383843 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c00cd80 con 0x7fc13c1001f0 2024-03-26T06:35:26.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.237+0000 7fc138ff9640 1 -- 172.21.15.38:0/24383843 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc12c0035f0 con 0x7fc13c1001f0 2024-03-26T06:35:26.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.237+0000 7fc138ff9640 1 -- 172.21.15.38:0/24383843 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12c010690 con 0x7fc13c1001f0 2024-03-26T06:35:26.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.237+0000 7fc141cc0640 1 -- 172.21.15.38:0/24383843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c19f570 con 0x7fc13c1001f0 2024-03-26T06:35:26.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.237+0000 7fc141cc0640 1 -- 172.21.15.38:0/24383843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c19fa90 con 0x7fc13c1001f0 2024-03-26T06:35:26.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.238+0000 7fc141cc0640 1 -- 172.21.15.38:0/24383843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c1075e0 con 0x7fc13c1001f0 2024-03-26T06:35:26.241 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.239+0000 7fc138ff9640 1 -- 172.21.15.38:0/24383843 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc12c0037a0 con 0x7fc13c1001f0 2024-03-26T06:35:26.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.240+0000 7fc138ff9640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc1240a2800 0x7fc1240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:26.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.240+0000 7fc13affd640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc1240a2800 0x7fc1240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:26.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.240+0000 7fc138ff9640 1 -- 172.21.15.38:0/24383843 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fc12c0ced40 con 0x7fc13c1001f0 2024-03-26T06:35:26.242 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.240+0000 7fc13affd640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc1240a2800 0x7fc1240a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc13002c7b0 tx=0x7fc130005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:26.250 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.248+0000 7fc138ff9640 1 -- 172.21.15.38:0/24383843 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc12c09c020 con 0x7fc13c1001f0 2024-03-26T06:35:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:26 smithi121 ceph-mon[36349]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:26 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3982587992' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-26T06:35:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:26 smithi121 ceph-mon[36349]: osdmap e83: 12 total, 12 up, 12 in 2024-03-26T06:35:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:26 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/982831581' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-26T06:35:26.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:26 smithi170 ceph-mon[36912]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:26.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:26 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3982587992' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-26T06:35:26.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:26 smithi170 ceph-mon[36912]: osdmap e83: 12 total, 12 up, 12 in 2024-03-26T06:35:26.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:26 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/982831581' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-26T06:35:26.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:26.476+0000 7fc141cc0640 1 -- 172.21.15.38:0/24383843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc13c061f90 con 0x7fc13c1001f0 2024-03-26T06:35:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:26 smithi038 ceph-mon[32265]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:26 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3982587992' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-26T06:35:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:26 smithi038 ceph-mon[32265]: osdmap e83: 12 total, 12 up, 12 in 2024-03-26T06:35:26.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:26 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/982831581' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-26T06:35:27.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.307+0000 7fc138ff9640 1 -- 172.21.15.38:0/24383843 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc12c09b920 con 0x7fc13c1001f0 2024-03-26T06:35:27.309 INFO:teuthology.orchestra.run.smithi038.stderr:new fs with metadata pool 2 and data pool 3 2024-03-26T06:35:27.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.310+0000 7fc141cc0640 1 -- 172.21.15.38:0/24383843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc1240a2800 msgr2=0x7fc1240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.310+0000 7fc141cc0640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc1240a2800 0x7fc1240a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc13002c7b0 tx=0x7fc130005fd0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.310+0000 7fc141cc0640 1 -- 172.21.15.38:0/24383843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc13c1001f0 msgr2=0x7fc13c068fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.310+0000 7fc141cc0640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc13c1001f0 0x7fc13c068fc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc12c00e9c0 tx=0x7fc12c00ee80 comp rx=0 tx=0).stop 2024-03-26T06:35:27.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.310+0000 7fc141cc0640 1 -- 172.21.15.38:0/24383843 shutdown_connections 2024-03-26T06:35:27.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.310+0000 7fc141cc0640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc1240a2800 0x7fc1240a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.310+0000 7fc141cc0640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc13c101da0 0x7fc13c19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.310+0000 7fc141cc0640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc13c1013e0 0x7fc13c19c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.310+0000 7fc141cc0640 1 --2- 172.21.15.38:0/24383843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc13c1001f0 0x7fc13c068fc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.310+0000 7fc141cc0640 1 -- 172.21.15.38:0/24383843 >> 172.21.15.38:0/24383843 conn(0x7fc13c0fb780 msgr2=0x7fc13c1047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:27.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.311+0000 7fc141cc0640 1 -- 172.21.15.38:0/24383843 shutdown_connections 2024-03-26T06:35:27.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.311+0000 7fc141cc0640 1 -- 172.21.15.38:0/24383843 wait complete. 2024-03-26T06:35:27.329 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:35:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/982831581' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-26T06:35:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: osdmap e84: 12 total, 12 up, 12 in 2024-03-26T06:35:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:35:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/24383843' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-26T06:35:27.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:27 smithi121 ceph-mon[36349]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-26T06:35:27.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/982831581' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-26T06:35:27.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: osdmap e84: 12 total, 12 up, 12 in 2024-03-26T06:35:27.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:27.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:35:27.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/24383843' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-26T06:35:27.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:27 smithi170 ceph-mon[36912]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-26T06:35:27.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.434+0000 7f3ac2f10640 1 -- 172.21.15.38:0/4021384592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3abc101dc0 msgr2=0x7f3abc108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.434+0000 7f3ac2f10640 1 --2- 172.21.15.38:0/4021384592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3abc101dc0 0x7f3abc108650 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4009ca0 tx=0x7f3ab402c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.434+0000 7f3ac2f10640 1 -- 172.21.15.38:0/4021384592 shutdown_connections 2024-03-26T06:35:27.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.434+0000 7f3ac2f10640 1 --2- 172.21.15.38:0/4021384592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3abc101dc0 0x7f3abc108650 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.434+0000 7f3ac2f10640 1 --2- 172.21.15.38:0/4021384592 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3abc101400 0x7f3abc101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.434+0000 7f3ac2f10640 1 --2- 172.21.15.38:0/4021384592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3abc100210 0x7f3abc100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.434+0000 7f3ac2f10640 1 -- 172.21.15.38:0/4021384592 >> 172.21.15.38:0/4021384592 conn(0x7f3abc0fb780 msgr2=0x7f3abc0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:27.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.434+0000 7f3ac2f10640 1 -- 172.21.15.38:0/4021384592 shutdown_connections 2024-03-26T06:35:27.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.434+0000 7f3ac2f10640 1 -- 172.21.15.38:0/4021384592 wait complete. 2024-03-26T06:35:27.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac2f10640 1 Processor -- start 2024-03-26T06:35:27.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac2f10640 1 -- start start 2024-03-26T06:35:27.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac2f10640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3abc100210 0x7f3abc1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:27.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac2f10640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3abc101400 0x7f3abc197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:27.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac2f10640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3abc101dc0 0x7f3abc19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:27.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac2f10640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc10bea0 con 0x7f3abc101dc0 2024-03-26T06:35:27.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac2f10640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc10bd20 con 0x7f3abc100210 2024-03-26T06:35:27.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac2f10640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3abc10c020 con 0x7f3abc101400 2024-03-26T06:35:27.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac14ad640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3abc101dc0 0x7f3abc19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac14ad640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3abc101dc0 0x7f3abc19eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55692/0 (socket says 172.21.15.38:55692) 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac14ad640 1 -- 172.21.15.38:0/2665109620 learned_addr learned my addr 172.21.15.38:0/2665109620 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ab3fff640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3abc101400 0x7f3abc197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.435+0000 7f3ac0cac640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3abc100210 0x7f3abc1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.436+0000 7f3ac14ad640 1 -- 172.21.15.38:0/2665109620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3abc101400 msgr2=0x7f3abc197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.436+0000 7f3ac14ad640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3abc101400 0x7f3abc197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.436+0000 7f3ac14ad640 1 -- 172.21.15.38:0/2665109620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3abc100210 msgr2=0x7f3abc1975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.436+0000 7f3ac14ad640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3abc100210 0x7f3abc1975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.436+0000 7f3ac14ad640 1 -- 172.21.15.38:0/2665109620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3abc19f210 con 0x7f3abc101dc0 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.436+0000 7f3ac14ad640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3abc101dc0 0x7f3abc19eba0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4005e30 tx=0x7f3ab4032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.436+0000 7f3ab1ffb640 1 -- 172.21.15.38:0/2665109620 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4041020 con 0x7f3abc101dc0 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.436+0000 7f3ab1ffb640 1 -- 172.21.15.38:0/2665109620 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ab4034cd0 con 0x7f3abc101dc0 2024-03-26T06:35:27.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.436+0000 7f3ac2f10640 1 -- 172.21.15.38:0/2665109620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3abc19f4a0 con 0x7f3abc101dc0 2024-03-26T06:35:27.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.436+0000 7f3ab1ffb640 1 -- 172.21.15.38:0/2665109620 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab4039040 con 0x7f3abc101dc0 2024-03-26T06:35:27.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.436+0000 7f3ac2f10640 1 -- 172.21.15.38:0/2665109620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3abc19f9e0 con 0x7f3abc101dc0 2024-03-26T06:35:27.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.437+0000 7f3ac2f10640 1 -- 172.21.15.38:0/2665109620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3abc19fc40 con 0x7f3abc101dc0 2024-03-26T06:35:27.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.438+0000 7f3ab1ffb640 1 -- 172.21.15.38:0/2665109620 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3ab4044020 con 0x7f3abc101dc0 2024-03-26T06:35:27.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.439+0000 7f3ab1ffb640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3aa40a2880 0x7f3aa40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:27.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.439+0000 7f3ac0cac640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3aa40a2880 0x7f3aa40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:27.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.439+0000 7f3ab1ffb640 1 -- 172.21.15.38:0/2665109620 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3ab40f08d0 con 0x7f3abc101dc0 2024-03-26T06:35:27.443 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.439+0000 7f3ac0cac640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3aa40a2880 0x7f3aa40a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3abc101260 tx=0x7f3ab800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:27.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.443+0000 7f3ab1ffb640 1 -- 172.21.15.38:0/2665109620 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3ab404b020 con 0x7f3abc101dc0 2024-03-26T06:35:27.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/982831581' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-26T06:35:27.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: osdmap e84: 12 total, 12 up, 12 in 2024-03-26T06:35:27.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:35:27.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:35:27.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:35:27.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/24383843' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-26T06:35:27.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:27 smithi038 ceph-mon[32265]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-26T06:35:27.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.627+0000 7f3ac2f10640 1 -- 172.21.15.38:0/2665109620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3abc19ffd0 con 0x7f3abc101dc0 2024-03-26T06:35:27.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.628+0000 7f3ab1ffb640 1 -- 172.21.15.38:0/2665109620 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22222 (secure 0 0 0) 0x7f3ab40dbe20 con 0x7f3abc101dc0 2024-03-26T06:35:27.631 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:27.631 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":85,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:35:26.477485+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:35:27.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 -- 172.21.15.38:0/2665109620 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3aa40a2880 msgr2=0x7f3aa40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3aa40a2880 0x7f3aa40a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3abc101260 tx=0x7f3ab800b390 comp rx=0 tx=0).stop 2024-03-26T06:35:27.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 -- 172.21.15.38:0/2665109620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3abc101dc0 msgr2=0x7f3abc19eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3abc101dc0 0x7f3abc19eba0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3ab4005e30 tx=0x7f3ab4032ec0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 -- 172.21.15.38:0/2665109620 shutdown_connections 2024-03-26T06:35:27.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3aa40a2880 0x7f3aa40a4d40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3abc101dc0 0x7f3abc19eba0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3abc101400 0x7f3abc197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 --2- 172.21.15.38:0/2665109620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3abc100210 0x7f3abc1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 -- 172.21.15.38:0/2665109620 >> 172.21.15.38:0/2665109620 conn(0x7f3abc0fb780 msgr2=0x7f3abc0fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:27.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 -- 172.21.15.38:0/2665109620 shutdown_connections 2024-03-26T06:35:27.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.630+0000 7f3ac2f10640 1 -- 172.21.15.38:0/2665109620 wait complete. 2024-03-26T06:35:27.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:35:27.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.744+0000 7f93b4c0c640 1 -- 172.21.15.38:0/3482955347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93b0101e70 msgr2=0x7f93b0104260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.744+0000 7f93b4c0c640 1 --2- 172.21.15.38:0/3482955347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93b0101e70 0x7f93b0104260 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f93a0009900 tx=0x7f93a002bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:27.749 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.744+0000 7f93b4c0c640 1 -- 172.21.15.38:0/3482955347 shutdown_connections 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.744+0000 7f93b4c0c640 1 --2- 172.21.15.38:0/3482955347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93b01070d0 0x7f93b01094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.744+0000 7f93b4c0c640 1 --2- 172.21.15.38:0/3482955347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93b01047a0 0x7f93b0106b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.744+0000 7f93b4c0c640 1 --2- 172.21.15.38:0/3482955347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93b0101e70 0x7f93b0104260 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.744+0000 7f93b4c0c640 1 -- 172.21.15.38:0/3482955347 >> 172.21.15.38:0/3482955347 conn(0x7f93b00fb820 msgr2=0x7f93b00fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.744+0000 7f93b4c0c640 1 -- 172.21.15.38:0/3482955347 shutdown_connections 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.744+0000 7f93b4c0c640 1 -- 172.21.15.38:0/3482955347 wait complete. 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93b4c0c640 1 Processor -- start 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93b4c0c640 1 -- start start 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93b4c0c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93b0101e70 0x7f93b01017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93b4c0c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93b01047a0 0x7f93b00ffe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93b4c0c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93b01070d0 0x7f93b0100360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93b4c0c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b010cfb0 con 0x7f93b0101e70 2024-03-26T06:35:27.750 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93b4c0c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b010ce30 con 0x7f93b01047a0 2024-03-26T06:35:27.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93b4c0c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b010d130 con 0x7f93b01070d0 2024-03-26T06:35:27.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93ae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93b0101e70 0x7f93b01017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:27.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93ae59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93b0101e70 0x7f93b01017e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55720/0 (socket says 172.21.15.38:55720) 2024-03-26T06:35:27.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93ae59c640 1 -- 172.21.15.38:0/967813983 learned_addr learned my addr 172.21.15.38:0/967813983 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:27.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93add9b640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93b01047a0 0x7f93b00ffe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:27.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93ae59c640 1 -- 172.21.15.38:0/967813983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93b01070d0 msgr2=0x7f93b0100360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93ae59c640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93b01070d0 0x7f93b0100360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93ae59c640 1 -- 172.21.15.38:0/967813983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93b01047a0 msgr2=0x7f93b00ffe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93ae59c640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93b01047a0 0x7f93b00ffe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93ae59c640 1 -- 172.21.15.38:0/967813983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b0100bf0 con 0x7f93b0101e70 2024-03-26T06:35:27.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93add9b640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93b01047a0 0x7f93b00ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:35:27.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.745+0000 7f93ae59c640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93b0101e70 0x7f93b01017e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f93a0009230 tx=0x7f93a0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:27.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.746+0000 7f939f7fe640 1 -- 172.21.15.38:0/967813983 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0041070 con 0x7f93b0101e70 2024-03-26T06:35:27.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.746+0000 7f939f7fe640 1 -- 172.21.15.38:0/967813983 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93a00054e0 con 0x7f93b0101e70 2024-03-26T06:35:27.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.746+0000 7f939f7fe640 1 -- 172.21.15.38:0/967813983 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0034440 con 0x7f93b0101e70 2024-03-26T06:35:27.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.746+0000 7f93b4c0c640 1 -- 172.21.15.38:0/967813983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b01a1710 con 0x7f93b0101e70 2024-03-26T06:35:27.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.746+0000 7f93b4c0c640 1 -- 172.21.15.38:0/967813983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b01a1cb0 con 0x7f93b0101e70 2024-03-26T06:35:27.752 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.747+0000 7f93b4c0c640 1 -- 172.21.15.38:0/967813983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b00fd310 con 0x7f93b0101e70 2024-03-26T06:35:27.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.748+0000 7f939f7fe640 1 -- 172.21.15.38:0/967813983 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f93a0044070 con 0x7f93b0101e70 2024-03-26T06:35:27.753 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.748+0000 7f939f7fe640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f93980a2880 0x7f93980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:27.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.748+0000 7f93add9b640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f93980a2880 0x7f93980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:27.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.748+0000 7f939f7fe640 1 -- 172.21.15.38:0/967813983 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f93a00f09d0 con 0x7f93b0101e70 2024-03-26T06:35:27.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.749+0000 7f93add9b640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f93980a2880 0x7f93980a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f93b0101550 tx=0x7f93a4005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:27.754 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.752+0000 7f939f7fe640 1 -- 172.21.15.38:0/967813983 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f93a00bdbc0 con 0x7f93b0101e70 2024-03-26T06:35:27.940 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.938+0000 7f93b4c0c640 1 -- 172.21.15.38:0/967813983 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93b002d040 con 0x7f93b0101e70 2024-03-26T06:35:27.941 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.939+0000 7f939f7fe640 1 -- 172.21.15.38:0/967813983 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22222 (secure 0 0 0) 0x7f93a00bd4c0 con 0x7f93b0101e70 2024-03-26T06:35:27.942 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:27.942 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":85,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:35:26.477485+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:35:27.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 -- 172.21.15.38:0/967813983 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f93980a2880 msgr2=0x7f93980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f93980a2880 0x7f93980a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f93b0101550 tx=0x7f93a4005d30 comp rx=0 tx=0).stop 2024-03-26T06:35:27.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 -- 172.21.15.38:0/967813983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93b0101e70 msgr2=0x7f93b01017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:27.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93b0101e70 0x7f93b01017e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f93a0009230 tx=0x7f93a0005dd0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 -- 172.21.15.38:0/967813983 shutdown_connections 2024-03-26T06:35:27.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f93980a2880 0x7f93980a4d40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f93b01070d0 0x7f93b0100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93b01047a0 0x7f93b00ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.944 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 --2- 172.21.15.38:0/967813983 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93b0101e70 0x7f93b01017e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:27.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 -- 172.21.15.38:0/967813983 >> 172.21.15.38:0/967813983 conn(0x7f93b00fb820 msgr2=0x7f93b0102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:27.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 -- 172.21.15.38:0/967813983 shutdown_connections 2024-03-26T06:35:27.945 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:27.941+0000 7f93b4c0c640 1 -- 172.21.15.38:0/967813983 wait complete. 2024-03-26T06:35:27.955 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-26T06:35:28.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.058+0000 7f0164174640 1 -- 172.21.15.38:0/1764434698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f015c101dc0 msgr2=0x7f015c108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:28.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.058+0000 7f0164174640 1 --2- 172.21.15.38:0/1764434698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f015c101dc0 0x7f015c108650 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0154009ca0 tx=0x7f015402c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:28.062 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.058+0000 7f0164174640 1 -- 172.21.15.38:0/1764434698 shutdown_connections 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.058+0000 7f0164174640 1 --2- 172.21.15.38:0/1764434698 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f015c101dc0 0x7f015c108650 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.058+0000 7f0164174640 1 --2- 172.21.15.38:0/1764434698 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f015c101400 0x7f015c101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.058+0000 7f0164174640 1 --2- 172.21.15.38:0/1764434698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f015c100210 0x7f015c100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.058+0000 7f0164174640 1 -- 172.21.15.38:0/1764434698 >> 172.21.15.38:0/1764434698 conn(0x7f015c0fb7a0 msgr2=0x7f015c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.058+0000 7f0164174640 1 -- 172.21.15.38:0/1764434698 shutdown_connections 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.058+0000 7f0164174640 1 -- 172.21.15.38:0/1764434698 wait complete. 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.058+0000 7f0164174640 1 Processor -- start 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.058+0000 7f0164174640 1 -- start start 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0164174640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f015c100210 0x7f015c1975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0164174640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f015c101400 0x7f015c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0164174640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f015c101dc0 0x7f015c19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:28.063 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0164174640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c10bea0 con 0x7f015c100210 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0164174640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c10bd20 con 0x7f015c101400 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0164174640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c10c020 con 0x7f015c101dc0 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0161f10640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f015c100210 0x7f015c1975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0161f10640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f015c100210 0x7f015c1975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55738/0 (socket says 172.21.15.38:55738) 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0161f10640 1 -- 172.21.15.38:0/2545087875 learned_addr learned my addr 172.21.15.38:0/2545087875 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0162711640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f015c101dc0 0x7f015c19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f016170f640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f015c101400 0x7f015c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0161f10640 1 -- 172.21.15.38:0/2545087875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f015c101dc0 msgr2=0x7f015c19eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0161f10640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f015c101dc0 0x7f015c19eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0161f10640 1 -- 172.21.15.38:0/2545087875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f015c101400 msgr2=0x7f015c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0161f10640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f015c101400 0x7f015c197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0161f10640 1 -- 172.21.15.38:0/2545087875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c19f1e0 con 0x7f015c100210 2024-03-26T06:35:28.064 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0162711640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f015c101dc0 0x7f015c19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:35:28.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f016170f640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f015c101400 0x7f015c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:35:28.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0161f10640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f015c100210 0x7f015c1975c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f015800d770 tx=0x7f015800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:28.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0152ffd640 1 -- 172.21.15.38:0/2545087875 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158017070 con 0x7f015c100210 2024-03-26T06:35:28.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0164174640 1 -- 172.21.15.38:0/2545087875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c19f4d0 con 0x7f015c100210 2024-03-26T06:35:28.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0152ffd640 1 -- 172.21.15.38:0/2545087875 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0158009c30 con 0x7f015c100210 2024-03-26T06:35:28.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0152ffd640 1 -- 172.21.15.38:0/2545087875 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0158003dd0 con 0x7f015c100210 2024-03-26T06:35:28.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.059+0000 7f0164174640 1 -- 172.21.15.38:0/2545087875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c19fa20 con 0x7f015c100210 2024-03-26T06:35:28.065 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.060+0000 7f0164174640 1 -- 172.21.15.38:0/2545087875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f015c19fc80 con 0x7f015c100210 2024-03-26T06:35:28.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.061+0000 7f0152ffd640 1 -- 172.21.15.38:0/2545087875 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0158003fb0 con 0x7f015c100210 2024-03-26T06:35:28.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.062+0000 7f0152ffd640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f01480a2800 0x7f01480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:28.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.062+0000 7f016170f640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f01480a2800 0x7f01480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:28.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.062+0000 7f0152ffd640 1 -- 172.21.15.38:0/2545087875 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f01580ce760 con 0x7f015c100210 2024-03-26T06:35:28.066 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.062+0000 7f016170f640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f01480a2800 0x7f01480a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f015c198ad0 tx=0x7f014c005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:28.068 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.066+0000 7f0152ffd640 1 -- 172.21.15.38:0/2545087875 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0158009de0 con 0x7f015c100210 2024-03-26T06:35:28.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:28.290+0000 7f0164174640 1 -- 172.21.15.38:0/2545087875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f015c19ff30 con 0x7f015c100210 2024-03-26T06:35:28.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:28 smithi038 ceph-mon[32265]: pgmap v284: 97 pgs: 11 creating+peering, 82 unknown, 4 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:28.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:28 smithi038 ceph-mon[32265]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T06:35:28.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:28 smithi038 ceph-mon[32265]: osdmap e85: 12 total, 12 up, 12 in 2024-03-26T06:35:28.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:28 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/24383843' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-26T06:35:28.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:28 smithi038 ceph-mon[32265]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-26T06:35:28.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:28 smithi038 ceph-mon[32265]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-26T06:35:28.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:28 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2665109620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:28.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:28 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] up:active 2024-03-26T06:35:28.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:28 smithi038 ceph-mon[32265]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-26T06:35:28.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:28 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/967813983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:28 smithi121 ceph-mon[36349]: pgmap v284: 97 pgs: 11 creating+peering, 82 unknown, 4 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:28 smithi121 ceph-mon[36349]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T06:35:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:28 smithi121 ceph-mon[36349]: osdmap e85: 12 total, 12 up, 12 in 2024-03-26T06:35:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:28 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/24383843' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-26T06:35:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:28 smithi121 ceph-mon[36349]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-26T06:35:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:28 smithi121 ceph-mon[36349]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-26T06:35:28.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:28 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2665109620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:28.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:28 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] up:active 2024-03-26T06:35:28.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:28 smithi121 ceph-mon[36349]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-26T06:35:28.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:28 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/967813983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:28.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:28 smithi170 ceph-mon[36912]: pgmap v284: 97 pgs: 11 creating+peering, 82 unknown, 4 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:28 smithi170 ceph-mon[36912]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T06:35:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:28 smithi170 ceph-mon[36912]: osdmap e85: 12 total, 12 up, 12 in 2024-03-26T06:35:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:28 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/24383843' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-26T06:35:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:28 smithi170 ceph-mon[36912]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-26T06:35:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:28 smithi170 ceph-mon[36912]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-26T06:35:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:28 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2665109620' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:28 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] up:active 2024-03-26T06:35:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:28 smithi170 ceph-mon[36912]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-26T06:35:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:28 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/967813983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:29.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.129+0000 7f0152ffd640 1 -- 172.21.15.38:0/2545087875 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) v1 ==== 118+0+0 (secure 0 0 0) 0x7f015809b950 con 0x7f015c100210 2024-03-26T06:35:29.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 -- 172.21.15.38:0/2545087875 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f01480a2800 msgr2=0x7f01480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:29.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f01480a2800 0x7f01480a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f015c198ad0 tx=0x7f014c005d30 comp rx=0 tx=0).stop 2024-03-26T06:35:29.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 -- 172.21.15.38:0/2545087875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f015c100210 msgr2=0x7f015c1975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:29.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f015c100210 0x7f015c1975c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f015800d770 tx=0x7f015800dc30 comp rx=0 tx=0).stop 2024-03-26T06:35:29.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 -- 172.21.15.38:0/2545087875 shutdown_connections 2024-03-26T06:35:29.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f01480a2800 0x7f01480a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:29.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f015c101dc0 0x7f015c19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:29.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f015c101400 0x7f015c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:29.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 --2- 172.21.15.38:0/2545087875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f015c100210 0x7f015c1975c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:29.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 -- 172.21.15.38:0/2545087875 >> 172.21.15.38:0/2545087875 conn(0x7f015c0fb7a0 msgr2=0x7f015c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:29.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 -- 172.21.15.38:0/2545087875 shutdown_connections 2024-03-26T06:35:29.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.132+0000 7f0164174640 1 -- 172.21.15.38:0/2545087875 wait complete. 2024-03-26T06:35:29.149 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-26T06:35:29.149 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-26T06:35:29.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.258+0000 7f1b536c0640 1 -- 172.21.15.38:0/2238508692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b4c101da0 msgr2=0x7f1b4c108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:29.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.258+0000 7f1b536c0640 1 --2- 172.21.15.38:0/2238508692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b4c101da0 0x7f1b4c108630 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009ca0 tx=0x7f1b4402c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:29.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.258+0000 7f1b536c0640 1 -- 172.21.15.38:0/2238508692 shutdown_connections 2024-03-26T06:35:29.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.258+0000 7f1b536c0640 1 --2- 172.21.15.38:0/2238508692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b4c101da0 0x7f1b4c108630 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:29.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.258+0000 7f1b536c0640 1 --2- 172.21.15.38:0/2238508692 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b4c1013e0 0x7f1b4c101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:29.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.258+0000 7f1b536c0640 1 --2- 172.21.15.38:0/2238508692 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b4c1001f0 0x7f1b4c100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:29.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.258+0000 7f1b536c0640 1 -- 172.21.15.38:0/2238508692 >> 172.21.15.38:0/2238508692 conn(0x7f1b4c0fb780 msgr2=0x7f1b4c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:29.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.258+0000 7f1b536c0640 1 -- 172.21.15.38:0/2238508692 shutdown_connections 2024-03-26T06:35:29.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.258+0000 7f1b536c0640 1 -- 172.21.15.38:0/2238508692 wait complete. 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b536c0640 1 Processor -- start 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b536c0640 1 -- start start 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b536c0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b4c1001f0 0x7f1b4c197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b536c0640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b4c1013e0 0x7f1b4c197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b536c0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b4c101da0 0x7f1b4c19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b536c0640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c10bec0 con 0x7f1b4c1001f0 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b536c0640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c10bd40 con 0x7f1b4c101da0 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b536c0640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c10c040 con 0x7f1b4c1013e0 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b5145c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b4c1001f0 0x7f1b4c197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b5145c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b4c1001f0 0x7f1b4c197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55756/0 (socket says 172.21.15.38:55756) 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b5145c640 1 -- 172.21.15.38:0/929046768 learned_addr learned my addr 172.21.15.38:0/929046768 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b51c5d640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b4c101da0 0x7f1b4c19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:29.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b50c5b640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b4c1013e0 0x7f1b4c197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:29.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b5145c640 1 -- 172.21.15.38:0/929046768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b4c1013e0 msgr2=0x7f1b4c197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:29.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b5145c640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b4c1013e0 0x7f1b4c197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:29.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b5145c640 1 -- 172.21.15.38:0/929046768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b4c101da0 msgr2=0x7f1b4c19ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:29.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b5145c640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b4c101da0 0x7f1b4c19ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:29.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b5145c640 1 -- 172.21.15.38:0/929046768 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c19f280 con 0x7f1b4c1001f0 2024-03-26T06:35:29.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.259+0000 7f1b5145c640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b4c1001f0 0x7f1b4c197660 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800d770 tx=0x7f1b4800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:29.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.260+0000 7f1b427fc640 1 -- 172.21.15.38:0/929046768 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48017070 con 0x7f1b4c1001f0 2024-03-26T06:35:29.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.260+0000 7f1b427fc640 1 -- 172.21.15.38:0/929046768 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b48009c30 con 0x7f1b4c1001f0 2024-03-26T06:35:29.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.260+0000 7f1b536c0640 1 -- 172.21.15.38:0/929046768 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4c19f570 con 0x7f1b4c1001f0 2024-03-26T06:35:29.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.260+0000 7f1b427fc640 1 -- 172.21.15.38:0/929046768 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48003dd0 con 0x7f1b4c1001f0 2024-03-26T06:35:29.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.260+0000 7f1b536c0640 1 -- 172.21.15.38:0/929046768 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4c19fac0 con 0x7f1b4c1001f0 2024-03-26T06:35:29.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.261+0000 7f1b536c0640 1 -- 172.21.15.38:0/929046768 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b4c19fd20 con 0x7f1b4c1001f0 2024-03-26T06:35:29.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.262+0000 7f1b427fc640 1 -- 172.21.15.38:0/929046768 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1b48003f90 con 0x7f1b4c1001f0 2024-03-26T06:35:29.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.262+0000 7f1b427fc640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1b380a2800 0x7f1b380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:29.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.262+0000 7f1b50c5b640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1b380a2800 0x7f1b380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:29.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.262+0000 7f1b427fc640 1 -- 172.21.15.38:0/929046768 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1b480ce770 con 0x7f1b4c1001f0 2024-03-26T06:35:29.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.263+0000 7f1b50c5b640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1b380a2800 0x7f1b380a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c198b70 tx=0x7f1b3c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:29.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.266+0000 7f1b427fc640 1 -- 172.21.15.38:0/929046768 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1b48009de0 con 0x7f1b4c1001f0 2024-03-26T06:35:29.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:29 smithi121 ceph-mon[36349]: osdmap e86: 12 total, 12 up, 12 in 2024-03-26T06:35:29.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:29 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2545087875' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-26T06:35:29.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:29 smithi170 ceph-mon[36912]: osdmap e86: 12 total, 12 up, 12 in 2024-03-26T06:35:29.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:29 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2545087875' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-26T06:35:29.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:29.488+0000 7f1b536c0640 1 -- 172.21.15.38:0/929046768 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f1b4c1a00e0 con 0x7f1b4c1001f0 2024-03-26T06:35:29.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:29 smithi038 ceph-mon[32265]: osdmap e86: 12 total, 12 up, 12 in 2024-03-26T06:35:29.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:29 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2545087875' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-26T06:35:30.141 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.138+0000 7f1b427fc640 1 -- 172.21.15.38:0/929046768 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v17) v1 ==== 105+0+0 (secure 0 0 0) 0x7f1b4809b960 con 0x7f1b4c1001f0 2024-03-26T06:35:30.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.142+0000 7f1b536c0640 1 -- 172.21.15.38:0/929046768 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1b380a2800 msgr2=0x7f1b380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:30.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.142+0000 7f1b536c0640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1b380a2800 0x7f1b380a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c198b70 tx=0x7f1b3c008040 comp rx=0 tx=0).stop 2024-03-26T06:35:30.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.142+0000 7f1b536c0640 1 -- 172.21.15.38:0/929046768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b4c1001f0 msgr2=0x7f1b4c197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:30.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.142+0000 7f1b536c0640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b4c1001f0 0x7f1b4c197660 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1b4800d770 tx=0x7f1b4800dc30 comp rx=0 tx=0).stop 2024-03-26T06:35:30.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.142+0000 7f1b536c0640 1 -- 172.21.15.38:0/929046768 shutdown_connections 2024-03-26T06:35:30.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.142+0000 7f1b536c0640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1b380a2800 0x7f1b380a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:30.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.142+0000 7f1b536c0640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1b4c101da0 0x7f1b4c19ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:30.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.142+0000 7f1b536c0640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1b4c1013e0 0x7f1b4c197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:30.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.142+0000 7f1b536c0640 1 --2- 172.21.15.38:0/929046768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b4c1001f0 0x7f1b4c197660 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:30.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.142+0000 7f1b536c0640 1 -- 172.21.15.38:0/929046768 >> 172.21.15.38:0/929046768 conn(0x7f1b4c0fb780 msgr2=0x7f1b4c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:30.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.142+0000 7f1b536c0640 1 -- 172.21.15.38:0/929046768 shutdown_connections 2024-03-26T06:35:30.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.143+0000 7f1b536c0640 1 -- 172.21.15.38:0/929046768 wait complete. 2024-03-26T06:35:30.162 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-26T06:35:30.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.271+0000 7f90d511b640 1 -- 172.21.15.38:0/2327251743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d0100200 msgr2=0x7f90d0100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:30.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.271+0000 7f90d511b640 1 --2- 172.21.15.38:0/2327251743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d0100200 0x7f90d0100620 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f90c0009900 tx=0x7f90c002bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:30.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.271+0000 7f90d511b640 1 -- 172.21.15.38:0/2327251743 shutdown_connections 2024-03-26T06:35:30.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.271+0000 7f90d511b640 1 --2- 172.21.15.38:0/2327251743 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d0101db0 0x7f90d0108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:30.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.271+0000 7f90d511b640 1 --2- 172.21.15.38:0/2327251743 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90d01013f0 0x7f90d0101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:30.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.271+0000 7f90d511b640 1 --2- 172.21.15.38:0/2327251743 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d0100200 0x7f90d0100620 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:30.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.271+0000 7f90d511b640 1 -- 172.21.15.38:0/2327251743 >> 172.21.15.38:0/2327251743 conn(0x7f90d00fb790 msgr2=0x7f90d00fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:30.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.271+0000 7f90d511b640 1 -- 172.21.15.38:0/2327251743 shutdown_connections 2024-03-26T06:35:30.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.271+0000 7f90d511b640 1 -- 172.21.15.38:0/2327251743 wait complete. 2024-03-26T06:35:30.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90d511b640 1 Processor -- start 2024-03-26T06:35:30.276 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90d511b640 1 -- start start 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90d511b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d0100200 0x7f90d0197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90d511b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d01013f0 0x7f90d0197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90d511b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90d0101db0 0x7f90d019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90d511b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d010be90 con 0x7f90d0100200 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90d511b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d010bd10 con 0x7f90d01013f0 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90d511b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d010c010 con 0x7f90d0101db0 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90ced9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d0100200 0x7f90d0197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90ced9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d0100200 0x7f90d0197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55776/0 (socket says 172.21.15.38:55776) 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90ced9d640 1 -- 172.21.15.38:0/3788994025 learned_addr learned my addr 172.21.15.38:0/3788994025 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90cf59e640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90d0101db0 0x7f90d019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90c7fff640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d01013f0 0x7f90d0197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:30.277 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90ced9d640 1 -- 172.21.15.38:0/3788994025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90d0101db0 msgr2=0x7f90d019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:30.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90ced9d640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90d0101db0 0x7f90d019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:30.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90ced9d640 1 -- 172.21.15.38:0/3788994025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d01013f0 msgr2=0x7f90d0197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:30.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90ced9d640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d01013f0 0x7f90d0197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:30.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90ced9d640 1 -- 172.21.15.38:0/3788994025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d019f2b0 con 0x7f90d0100200 2024-03-26T06:35:30.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.272+0000 7f90c7fff640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d01013f0 0x7f90d0197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:30.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.273+0000 7f90ced9d640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d0100200 0x7f90d0197690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f90c0005ab0 tx=0x7f90c0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:30.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.273+0000 7f90ccd99640 1 -- 172.21.15.38:0/3788994025 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c0041070 con 0x7f90d0100200 2024-03-26T06:35:30.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.273+0000 7f90d511b640 1 -- 172.21.15.38:0/3788994025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d019f540 con 0x7f90d0100200 2024-03-26T06:35:30.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.273+0000 7f90ccd99640 1 -- 172.21.15.38:0/3788994025 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90c00054e0 con 0x7f90d0100200 2024-03-26T06:35:30.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.273+0000 7f90ccd99640 1 -- 172.21.15.38:0/3788994025 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c0034440 con 0x7f90d0100200 2024-03-26T06:35:30.278 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.273+0000 7f90d511b640 1 -- 172.21.15.38:0/3788994025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d019fa30 con 0x7f90d0100200 2024-03-26T06:35:30.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.274+0000 7f90d511b640 1 -- 172.21.15.38:0/3788994025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d019fc90 con 0x7f90d0100200 2024-03-26T06:35:30.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.275+0000 7f90ccd99640 1 -- 172.21.15.38:0/3788994025 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f90c0044070 con 0x7f90d0100200 2024-03-26T06:35:30.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.275+0000 7f90ccd99640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f90bc0a2800 0x7f90bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:30.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.276+0000 7f90c7fff640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f90bc0a2800 0x7f90bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:30.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.276+0000 7f90ccd99640 1 -- 172.21.15.38:0/3788994025 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f90c00f07a0 con 0x7f90d0100200 2024-03-26T06:35:30.280 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.276+0000 7f90c7fff640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f90bc0a2800 0x7f90bc0a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f90b80058c0 tx=0x7f90b8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:30.281 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.279+0000 7f90ccd99640 1 -- 172.21.15.38:0/3788994025 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f90c00bd8f0 con 0x7f90d0100200 2024-03-26T06:35:30.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:30 smithi170 ceph-mon[36912]: pgmap v287: 97 pgs: 26 creating+peering, 53 unknown, 18 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:30.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:30 smithi170 ceph-mon[36912]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-26T06:35:30.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:30 smithi170 ceph-mon[36912]: Cluster is now healthy 2024-03-26T06:35:30.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:30 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2545087875' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-26T06:35:30.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:30 smithi170 ceph-mon[36912]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-26T06:35:30.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:30 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/929046768' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-26T06:35:30.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:30.499+0000 7f90d511b640 1 -- 172.21.15.38:0/3788994025 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f90d019fe60 con 0x7f90d0100200 2024-03-26T06:35:30.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:30 smithi038 ceph-mon[32265]: pgmap v287: 97 pgs: 26 creating+peering, 53 unknown, 18 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:30.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:30 smithi038 ceph-mon[32265]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-26T06:35:30.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:30 smithi038 ceph-mon[32265]: Cluster is now healthy 2024-03-26T06:35:30.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:30 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2545087875' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-26T06:35:30.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:30 smithi038 ceph-mon[32265]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-26T06:35:30.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:30 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/929046768' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-26T06:35:30.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:30 smithi121 ceph-mon[36349]: pgmap v287: 97 pgs: 26 creating+peering, 53 unknown, 18 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:35:30.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:30 smithi121 ceph-mon[36349]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-26T06:35:30.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:30 smithi121 ceph-mon[36349]: Cluster is now healthy 2024-03-26T06:35:30.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:30 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2545087875' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-26T06:35:30.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:30 smithi121 ceph-mon[36349]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-26T06:35:30.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:30 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/929046768' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-26T06:35:31.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.152+0000 7f90ccd99640 1 -- 172.21.15.38:0/3788994025 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) v1 ==== 121+0+0 (secure 0 0 0) 0x7f90c00bd1f0 con 0x7f90d0100200 2024-03-26T06:35:31.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.155+0000 7f90d511b640 1 -- 172.21.15.38:0/3788994025 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f90bc0a2800 msgr2=0x7f90bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:31.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.155+0000 7f90d511b640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f90bc0a2800 0x7f90bc0a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f90b80058c0 tx=0x7f90b8005850 comp rx=0 tx=0).stop 2024-03-26T06:35:31.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.155+0000 7f90d511b640 1 -- 172.21.15.38:0/3788994025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d0100200 msgr2=0x7f90d0197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:31.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.155+0000 7f90d511b640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d0100200 0x7f90d0197690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f90c0005ab0 tx=0x7f90c0005dc0 comp rx=0 tx=0).stop 2024-03-26T06:35:31.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.155+0000 7f90d511b640 1 -- 172.21.15.38:0/3788994025 shutdown_connections 2024-03-26T06:35:31.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.155+0000 7f90d511b640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f90bc0a2800 0x7f90bc0a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:31.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.155+0000 7f90d511b640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f90d0101db0 0x7f90d019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:31.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.155+0000 7f90d511b640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f90d01013f0 0x7f90d0197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:31.158 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.155+0000 7f90d511b640 1 --2- 172.21.15.38:0/3788994025 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f90d0100200 0x7f90d0197690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:31.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.155+0000 7f90d511b640 1 -- 172.21.15.38:0/3788994025 >> 172.21.15.38:0/3788994025 conn(0x7f90d00fb790 msgr2=0x7f90d00698f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:31.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.156+0000 7f90d511b640 1 -- 172.21.15.38:0/3788994025 shutdown_connections 2024-03-26T06:35:31.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.156+0000 7f90d511b640 1 -- 172.21.15.38:0/3788994025 wait complete. 2024-03-26T06:35:31.175 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-26T06:35:31.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.284+0000 7fc5683ab640 1 -- 172.21.15.38:0/3087468499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc560102f70 msgr2=0x7fc560109800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:31.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.284+0000 7fc5683ab640 1 --2- 172.21.15.38:0/3087468499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc560102f70 0x7fc560109800 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc550009cd0 tx=0x7fc55002c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:31.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.284+0000 7fc5683ab640 1 -- 172.21.15.38:0/3087468499 shutdown_connections 2024-03-26T06:35:31.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.284+0000 7fc5683ab640 1 --2- 172.21.15.38:0/3087468499 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc560102f70 0x7fc560109800 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:31.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.284+0000 7fc5683ab640 1 --2- 172.21.15.38:0/3087468499 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5601025b0 0x7fc560102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:31.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.284+0000 7fc5683ab640 1 --2- 172.21.15.38:0/3087468499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc5600ff5e0 0x7fc5600ffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:31.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.284+0000 7fc5683ab640 1 -- 172.21.15.38:0/3087468499 >> 172.21.15.38:0/3087468499 conn(0x7fc5600faef0 msgr2=0x7fc5600fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:31.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.284+0000 7fc5683ab640 1 -- 172.21.15.38:0/3087468499 shutdown_connections 2024-03-26T06:35:31.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.284+0000 7fc5683ab640 1 -- 172.21.15.38:0/3087468499 wait complete. 2024-03-26T06:35:31.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.284+0000 7fc5683ab640 1 Processor -- start 2024-03-26T06:35:31.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.284+0000 7fc5683ab640 1 -- start start 2024-03-26T06:35:31.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc5683ab640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5600ff5e0 0x7fc5601976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:31.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc5683ab640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5601025b0 0x7fc560197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:31.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc5683ab640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc560102f70 0x7fc56019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc5683ab640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56010d0b0 con 0x7fc5600ff5e0 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc5683ab640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56010cf30 con 0x7fc560102f70 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc5683ab640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56010d230 con 0x7fc5601025b0 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc566147640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5600ff5e0 0x7fc5601976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc566147640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5600ff5e0 0x7fc5601976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55784/0 (socket says 172.21.15.38:55784) 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc566147640 1 -- 172.21.15.38:0/1952208765 learned_addr learned my addr 172.21.15.38:0/1952208765 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc565946640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5601025b0 0x7fc560197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc566948640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc560102f70 0x7fc56019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc566147640 1 -- 172.21.15.38:0/1952208765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5601025b0 msgr2=0x7fc560197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc566147640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5601025b0 0x7fc560197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc566147640 1 -- 172.21.15.38:0/1952208765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc560102f70 msgr2=0x7fc56019ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc566147640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc560102f70 0x7fc56019ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc566147640 1 -- 172.21.15.38:0/1952208765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56019f2f0 con 0x7fc5600ff5e0 2024-03-26T06:35:31.291 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc566948640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc560102f70 0x7fc56019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:35:31.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc566147640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5600ff5e0 0x7fc5601976d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc55800b450 tx=0x7fc55800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:31.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc55f7fe640 1 -- 172.21.15.38:0/1952208765 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc558015020 con 0x7fc5600ff5e0 2024-03-26T06:35:31.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc55f7fe640 1 -- 172.21.15.38:0/1952208765 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc558011d70 con 0x7fc5600ff5e0 2024-03-26T06:35:31.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc5683ab640 1 -- 172.21.15.38:0/1952208765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56019f5e0 con 0x7fc5600ff5e0 2024-03-26T06:35:31.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc55f7fe640 1 -- 172.21.15.38:0/1952208765 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55800f340 con 0x7fc5600ff5e0 2024-03-26T06:35:31.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.285+0000 7fc5683ab640 1 -- 172.21.15.38:0/1952208765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56019fbb0 con 0x7fc5600ff5e0 2024-03-26T06:35:31.292 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.286+0000 7fc5683ab640 1 -- 172.21.15.38:0/1952208765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5601086e0 con 0x7fc5600ff5e0 2024-03-26T06:35:31.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.288+0000 7fc55f7fe640 1 -- 172.21.15.38:0/1952208765 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc558018020 con 0x7fc5600ff5e0 2024-03-26T06:35:31.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.288+0000 7fc55f7fe640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc5480a2800 0x7fc5480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:31.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.288+0000 7fc565946640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc5480a2800 0x7fc5480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:31.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.288+0000 7fc55f7fe640 1 -- 172.21.15.38:0/1952208765 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fc5580ce510 con 0x7fc5600ff5e0 2024-03-26T06:35:31.293 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.289+0000 7fc565946640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc5480a2800 0x7fc5480a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc560198be0 tx=0x7fc554008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:31.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.293+0000 7fc55f7fe640 1 -- 172.21.15.38:0/1952208765 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc55801f020 con 0x7fc5600ff5e0 2024-03-26T06:35:31.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:31 smithi170 ceph-mon[36912]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-26T06:35:31.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:31 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/929046768' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-26T06:35:31.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:31 smithi170 ceph-mon[36912]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-26T06:35:31.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:31 smithi170 ceph-mon[36912]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-26T06:35:31.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:31 smithi170 ceph-mon[36912]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-26T06:35:31.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:31 smithi170 ceph-mon[36912]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-26T06:35:31.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:31 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3788994025' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-26T06:35:31.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:31.515+0000 7fc5683ab640 1 -- 172.21.15.38:0/1952208765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fc56002ce70 con 0x7fc5600ff5e0 2024-03-26T06:35:31.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:31 smithi038 ceph-mon[32265]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-26T06:35:31.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:31 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/929046768' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-26T06:35:31.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:31 smithi038 ceph-mon[32265]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-26T06:35:31.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:31 smithi038 ceph-mon[32265]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-26T06:35:31.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:31 smithi038 ceph-mon[32265]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-26T06:35:31.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:31 smithi038 ceph-mon[32265]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-26T06:35:31.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:31 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3788994025' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-26T06:35:31.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:31 smithi121 ceph-mon[36349]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-26T06:35:31.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:31 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/929046768' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-26T06:35:31.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:31 smithi121 ceph-mon[36349]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-26T06:35:31.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:31 smithi121 ceph-mon[36349]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-26T06:35:31.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:31 smithi121 ceph-mon[36349]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-26T06:35:31.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:31 smithi121 ceph-mon[36349]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-26T06:35:31.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:31 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3788994025' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-26T06:35:32.164 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.162+0000 7fc55f7fe640 1 -- 172.21.15.38:0/1952208765 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) v1 ==== 115+0+0 (secure 0 0 0) 0x7fc55800d040 con 0x7fc5600ff5e0 2024-03-26T06:35:32.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 -- 172.21.15.38:0/1952208765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc5480a2800 msgr2=0x7fc5480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.168 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc5480a2800 0x7fc5480a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc560198be0 tx=0x7fc554008040 comp rx=0 tx=0).stop 2024-03-26T06:35:32.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 -- 172.21.15.38:0/1952208765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5600ff5e0 msgr2=0x7fc5601976d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5600ff5e0 0x7fc5601976d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc55800b450 tx=0x7fc55800b910 comp rx=0 tx=0).stop 2024-03-26T06:35:32.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 -- 172.21.15.38:0/1952208765 shutdown_connections 2024-03-26T06:35:32.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc5480a2800 0x7fc5480a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc560102f70 0x7fc56019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc5601025b0 0x7fc560197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 --2- 172.21.15.38:0/1952208765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc5600ff5e0 0x7fc5601976d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 -- 172.21.15.38:0/1952208765 >> 172.21.15.38:0/1952208765 conn(0x7fc5600faef0 msgr2=0x7fc5601058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:32.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 -- 172.21.15.38:0/1952208765 shutdown_connections 2024-03-26T06:35:32.169 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.166+0000 7fc5683ab640 1 -- 172.21.15.38:0/1952208765 wait complete. 2024-03-26T06:35:32.184 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-26T06:35:32.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-26T06:35:32.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.293+0000 7fed03ca4640 1 -- 172.21.15.38:0/2350441112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecfc101db0 msgr2=0x7fecfc108640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.293+0000 7fed03ca4640 1 --2- 172.21.15.38:0/2350441112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecfc101db0 0x7fecfc108640 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fecf4009ca0 tx=0x7fecf402c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.293+0000 7fed03ca4640 1 -- 172.21.15.38:0/2350441112 shutdown_connections 2024-03-26T06:35:32.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.293+0000 7fed03ca4640 1 --2- 172.21.15.38:0/2350441112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecfc101db0 0x7fecfc108640 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.293+0000 7fed03ca4640 1 --2- 172.21.15.38:0/2350441112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecfc1013f0 0x7fecfc101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.293+0000 7fed03ca4640 1 --2- 172.21.15.38:0/2350441112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecfc100200 0x7fecfc100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.293+0000 7fed03ca4640 1 -- 172.21.15.38:0/2350441112 >> 172.21.15.38:0/2350441112 conn(0x7fecfc0fb790 msgr2=0x7fecfc0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.293+0000 7fed03ca4640 1 -- 172.21.15.38:0/2350441112 shutdown_connections 2024-03-26T06:35:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.293+0000 7fed03ca4640 1 -- 172.21.15.38:0/2350441112 wait complete. 2024-03-26T06:35:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed03ca4640 1 Processor -- start 2024-03-26T06:35:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed03ca4640 1 -- start start 2024-03-26T06:35:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed03ca4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecfc100200 0x7fecfc06a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed03ca4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecfc1013f0 0x7fecfc19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed03ca4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecfc101db0 0x7fecfc19e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed03ca4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc10be90 con 0x7fecfc100200 2024-03-26T06:35:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed03ca4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc10bd10 con 0x7fecfc1013f0 2024-03-26T06:35:32.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed03ca4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecfc10c010 con 0x7fecfc101db0 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed01a40640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecfc100200 0x7fecfc06a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed01a40640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecfc100200 0x7fecfc06a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55794/0 (socket says 172.21.15.38:55794) 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed01a40640 1 -- 172.21.15.38:0/2624228327 learned_addr learned my addr 172.21.15.38:0/2624228327 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed02241640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecfc101db0 0x7fecfc19e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed0123f640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecfc1013f0 0x7fecfc19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed01a40640 1 -- 172.21.15.38:0/2624228327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecfc101db0 msgr2=0x7fecfc19e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed01a40640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecfc101db0 0x7fecfc19e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed01a40640 1 -- 172.21.15.38:0/2624228327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecfc1013f0 msgr2=0x7fecfc19c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed01a40640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecfc1013f0 0x7fecfc19c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.294+0000 7fed01a40640 1 -- 172.21.15.38:0/2624228327 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecfc19f1c0 con 0x7fecfc100200 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.295+0000 7fed01a40640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecfc100200 0x7fecfc06a450 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fecf800c970 tx=0x7fecf800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.295+0000 7fecf2ffd640 1 -- 172.21.15.38:0/2624228327 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8004220 con 0x7fecfc100200 2024-03-26T06:35:32.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.295+0000 7fecf2ffd640 1 -- 172.21.15.38:0/2624228327 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecf80043c0 con 0x7fecfc100200 2024-03-26T06:35:32.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.295+0000 7fed03ca4640 1 -- 172.21.15.38:0/2624228327 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecfc19f4b0 con 0x7fecfc100200 2024-03-26T06:35:32.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.295+0000 7fecf2ffd640 1 -- 172.21.15.38:0/2624228327 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf8004a00 con 0x7fecfc100200 2024-03-26T06:35:32.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.295+0000 7fed03ca4640 1 -- 172.21.15.38:0/2624228327 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecfc19f9b0 con 0x7fecfc100200 2024-03-26T06:35:32.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.296+0000 7fed03ca4640 1 -- 172.21.15.38:0/2624228327 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecfc107520 con 0x7fecfc100200 2024-03-26T06:35:32.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.297+0000 7fecf2ffd640 1 -- 172.21.15.38:0/2624228327 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fecf80079b0 con 0x7fecfc100200 2024-03-26T06:35:32.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.297+0000 7fecf2ffd640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fece80a2800 0x7fece80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:32.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.297+0000 7fed0123f640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fece80a2800 0x7fece80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:32.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.297+0000 7fecf2ffd640 1 -- 172.21.15.38:0/2624228327 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fecf80ce740 con 0x7fecfc100200 2024-03-26T06:35:32.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.298+0000 7fed0123f640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fece80a2800 0x7fece80a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fecfc06a1c0 tx=0x7fecec00b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:32.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.302+0000 7fecf2ffd640 1 -- 172.21.15.38:0/2624228327 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fecf8014030 con 0x7fecfc100200 2024-03-26T06:35:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:32 smithi170 ceph-mon[36912]: pgmap v288: 97 pgs: 26 creating+peering, 14 unknown, 57 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 596 B/s wr, 4 op/s 2024-03-26T06:35:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:32 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3788994025' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-26T06:35:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:32 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] up:active 2024-03-26T06:35:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:32 smithi170 ceph-mon[36912]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-26T06:35:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:32 smithi170 ceph-mon[36912]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-26T06:35:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:32 smithi170 ceph-mon[36912]: Cluster is now healthy 2024-03-26T06:35:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:32 smithi170 ceph-mon[36912]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-26T06:35:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:32 smithi170 ceph-mon[36912]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-26T06:35:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:32 smithi170 ceph-mon[36912]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-26T06:35:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:32 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1952208765' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-26T06:35:32.511 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.509+0000 7fed03ca4640 1 -- 172.21.15.38:0/2624228327 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fecfc061730 con 0x7fece80a2800 2024-03-26T06:35:32.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-mon[32265]: pgmap v288: 97 pgs: 26 creating+peering, 14 unknown, 57 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 596 B/s wr, 4 op/s 2024-03-26T06:35:32.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3788994025' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-26T06:35:32.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] up:active 2024-03-26T06:35:32.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-mon[32265]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-26T06:35:32.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-mon[32265]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-26T06:35:32.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-mon[32265]: Cluster is now healthy 2024-03-26T06:35:32.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-mon[32265]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-26T06:35:32.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-mon[32265]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-26T06:35:32.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-mon[32265]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-26T06:35:32.518 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1952208765' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-26T06:35:32.534 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.532+0000 7fecf2ffd640 1 -- 172.21.15.38:0/2624228327 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fecfc061730 con 0x7fece80a2800 2024-03-26T06:35:32.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.533+0000 7fed03ca4640 1 -- 172.21.15.38:0/2624228327 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fece80a2800 msgr2=0x7fece80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.533+0000 7fed03ca4640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fece80a2800 0x7fece80a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fecfc06a1c0 tx=0x7fecec00b590 comp rx=0 tx=0).stop 2024-03-26T06:35:32.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.533+0000 7fed03ca4640 1 -- 172.21.15.38:0/2624228327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecfc100200 msgr2=0x7fecfc06a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.533+0000 7fed03ca4640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecfc100200 0x7fecfc06a450 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fecf800c970 tx=0x7fecf800ce30 comp rx=0 tx=0).stop 2024-03-26T06:35:32.536 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.533+0000 7fed03ca4640 1 -- 172.21.15.38:0/2624228327 shutdown_connections 2024-03-26T06:35:32.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.534+0000 7fed03ca4640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fece80a2800 0x7fece80a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.534+0000 7fed03ca4640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fecfc101db0 0x7fecfc19e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.534+0000 7fed03ca4640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fecfc1013f0 0x7fecfc19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.534+0000 7fed03ca4640 1 --2- 172.21.15.38:0/2624228327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecfc100200 0x7fecfc06a450 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.534+0000 7fed03ca4640 1 -- 172.21.15.38:0/2624228327 >> 172.21.15.38:0/2624228327 conn(0x7fecfc0fb790 msgr2=0x7fecfc1046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:32.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.534+0000 7fed03ca4640 1 -- 172.21.15.38:0/2624228327 shutdown_connections 2024-03-26T06:35:32.537 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.534+0000 7fed03ca4640 1 -- 172.21.15.38:0/2624228327 wait complete. 2024-03-26T06:35:32.547 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-26T06:35:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-mon[36349]: pgmap v288: 97 pgs: 26 creating+peering, 14 unknown, 57 active+clean; 577 KiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 596 B/s wr, 4 op/s 2024-03-26T06:35:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3788994025' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-26T06:35:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] up:active 2024-03-26T06:35:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-mon[36349]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-26T06:35:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-mon[36349]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-26T06:35:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-mon[36349]: Cluster is now healthy 2024-03-26T06:35:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-mon[36349]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-26T06:35:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-mon[36349]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-03-26T06:35:32.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-mon[36349]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-03-26T06:35:32.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1952208765' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-26T06:35:32.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.651+0000 7f4592124640 1 -- 172.21.15.38:0/3261787960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f458c106fd0 msgr2=0x7f458c109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.651+0000 7f4592124640 1 --2- 172.21.15.38:0/3261787960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f458c106fd0 0x7f458c109420 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f4584009ca0 tx=0x7f458402c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.651+0000 7f4592124640 1 -- 172.21.15.38:0/3261787960 shutdown_connections 2024-03-26T06:35:32.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.651+0000 7f4592124640 1 --2- 172.21.15.38:0/3261787960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f458c106fd0 0x7f458c109420 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.651+0000 7f4592124640 1 --2- 172.21.15.38:0/3261787960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f458c1046a0 0x7f458c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.651+0000 7f4592124640 1 --2- 172.21.15.38:0/3261787960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f458c101d70 0x7f458c104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.651+0000 7f4592124640 1 -- 172.21.15.38:0/3261787960 >> 172.21.15.38:0/3261787960 conn(0x7f458c0fb780 msgr2=0x7f458c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:32.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.651+0000 7f4592124640 1 -- 172.21.15.38:0/3261787960 shutdown_connections 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.651+0000 7f4592124640 1 -- 172.21.15.38:0/3261787960 wait complete. 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.652+0000 7f4592124640 1 Processor -- start 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.652+0000 7f4592124640 1 -- start start 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.652+0000 7f4592124640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f458c101d70 0x7f458c1016e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.652+0000 7f4592124640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f458c1046a0 0x7f458c0ffd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.652+0000 7f4592124640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f458c106fd0 0x7f458c100260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.652+0000 7f4592124640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c10d060 con 0x7f458c1046a0 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.652+0000 7f4592124640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c10cee0 con 0x7f458c101d70 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.652+0000 7f4592124640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c10d1e0 con 0x7f458c106fd0 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.652+0000 7f458affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f458c1046a0 0x7f458c0ffd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.652+0000 7f458affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f458c1046a0 0x7f458c0ffd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55816/0 (socket says 172.21.15.38:55816) 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f458affd640 1 -- 172.21.15.38:0/865087109 learned_addr learned my addr 172.21.15.38:0/865087109 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:32.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f458b7fe640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f458c101d70 0x7f458c1016e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f458bfff640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f458c106fd0 0x7f458c100260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f458affd640 1 -- 172.21.15.38:0/865087109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f458c106fd0 msgr2=0x7f458c100260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f458affd640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f458c106fd0 0x7f458c100260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f458affd640 1 -- 172.21.15.38:0/865087109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f458c101d70 msgr2=0x7f458c1016e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f458affd640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f458c101d70 0x7f458c1016e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f458affd640 1 -- 172.21.15.38:0/865087109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458c100af0 con 0x7f458c1046a0 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f458bfff640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f458c106fd0 0x7f458c100260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f458b7fe640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f458c101d70 0x7f458c1016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f458affd640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f458c1046a0 0x7f458c0ffd20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f458000b450 tx=0x7f458000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f4588ff9640 1 -- 172.21.15.38:0/865087109 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4580015020 con 0x7f458c1046a0 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f4588ff9640 1 -- 172.21.15.38:0/865087109 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4580011d70 con 0x7f458c1046a0 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f4592124640 1 -- 172.21.15.38:0/865087109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458c1a17c0 con 0x7f458c1046a0 2024-03-26T06:35:32.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f4588ff9640 1 -- 172.21.15.38:0/865087109 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458000f340 con 0x7f458c1046a0 2024-03-26T06:35:32.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.653+0000 7f4592124640 1 -- 172.21.15.38:0/865087109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458c1a1d90 con 0x7f458c1046a0 2024-03-26T06:35:32.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.654+0000 7f4592124640 1 -- 172.21.15.38:0/865087109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458c0fd270 con 0x7f458c1046a0 2024-03-26T06:35:32.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.655+0000 7f4588ff9640 1 -- 172.21.15.38:0/865087109 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4580018020 con 0x7f458c1046a0 2024-03-26T06:35:32.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.656+0000 7f4588ff9640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f45740a2800 0x7f45740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:32.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.656+0000 7f458b7fe640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f45740a2800 0x7f45740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:32.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.656+0000 7f4588ff9640 1 -- 172.21.15.38:0/865087109 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f45800ce4c0 con 0x7f458c1046a0 2024-03-26T06:35:32.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.656+0000 7f458b7fe640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f45740a2800 0x7f45740a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f457c009930 tx=0x7f457c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:32.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.660+0000 7f4588ff9640 1 -- 172.21.15.38:0/865087109 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f458001f020 con 0x7f458c1046a0 2024-03-26T06:35:32.833 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:35:32.513+0000 7f0f5ebf2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:35:32.833 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:35:32.513+0000 7f0f5ebf2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:35:32.833 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:35:32.513+0000 7f0f5ebf2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:35:32.833 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:35:32.513+0000 7f0f5ebf2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:35:32.833 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:35:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:35:32.513+0000 7f0f5ebf2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:35:32.872 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.870+0000 7f4592124640 1 -- 172.21.15.38:0/865087109 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f458c0617c0 con 0x7f45740a2800 2024-03-26T06:35:32.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.891+0000 7f4588ff9640 1 -- 172.21.15.38:0/865087109 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f458c0617c0 con 0x7f45740a2800 2024-03-26T06:35:32.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 -- 172.21.15.38:0/865087109 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f45740a2800 msgr2=0x7f45740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f45740a2800 0x7f45740a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f457c009930 tx=0x7f457c008040 comp rx=0 tx=0).stop 2024-03-26T06:35:32.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 -- 172.21.15.38:0/865087109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f458c1046a0 msgr2=0x7f458c0ffd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:32.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f458c1046a0 0x7f458c0ffd20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f458000b450 tx=0x7f458000b910 comp rx=0 tx=0).stop 2024-03-26T06:35:32.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 -- 172.21.15.38:0/865087109 shutdown_connections 2024-03-26T06:35:32.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f45740a2800 0x7f45740a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.896 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f458c106fd0 0x7f458c100260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f458c1046a0 0x7f458c0ffd20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 --2- 172.21.15.38:0/865087109 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f458c101d70 0x7f458c1016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:32.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 -- 172.21.15.38:0/865087109 >> 172.21.15.38:0/865087109 conn(0x7f458c0fb780 msgr2=0x7f458c1026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:32.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 -- 172.21.15.38:0/865087109 shutdown_connections 2024-03-26T06:35:32.897 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:32.893+0000 7f4592124640 1 -- 172.21.15.38:0/865087109 wait complete. 2024-03-26T06:35:32.906 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-26T06:35:33.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 -- 172.21.15.38:0/2735530221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0e8101310 msgr2=0x7fb0e8101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 --2- 172.21.15.38:0/2735530221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0e8101310 0x7fb0e8101730 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009900 tx=0x7fb0d802bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:33.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 -- 172.21.15.38:0/2735530221 shutdown_connections 2024-03-26T06:35:33.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 --2- 172.21.15.38:0/2735530221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e8102ec0 0x7fb0e8109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 --2- 172.21.15.38:0/2735530221 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e8102500 0x7fb0e8102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 --2- 172.21.15.38:0/2735530221 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0e8101310 0x7fb0e8101730 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 -- 172.21.15.38:0/2735530221 >> 172.21.15.38:0/2735530221 conn(0x7fb0e80fc8a0 msgr2=0x7fb0e80fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 -- 172.21.15.38:0/2735530221 shutdown_connections 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 -- 172.21.15.38:0/2735530221 wait complete. 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 Processor -- start 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 -- start start 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e8101310 0x7fb0e80750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e8102500 0x7fb0e80736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.012+0000 7fb0ed345640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0e8102ec0 0x7fb0e8073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0ed345640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e810cfa0 con 0x7fb0e8102ec0 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0ed345640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e810ce20 con 0x7fb0e8102500 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0ed345640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e810d120 con 0x7fb0e8101310 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0e8102ec0 0x7fb0e8073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0e8102ec0 0x7fb0e8073c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55826/0 (socket says 172.21.15.38:55826) 2024-03-26T06:35:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e77fe640 1 -- 172.21.15.38:0/1554172999 learned_addr learned my addr 172.21.15.38:0/1554172999 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:33.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e6ffd640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e8101310 0x7fb0e80750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e77fe640 1 -- 172.21.15.38:0/1554172999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e8101310 msgr2=0x7fb0e80750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e77fe640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e8101310 0x7fb0e80750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e67fc640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e8102500 0x7fb0e80736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e77fe640 1 -- 172.21.15.38:0/1554172999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e8102500 msgr2=0x7fb0e80736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e77fe640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e8102500 0x7fb0e80736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e77fe640 1 -- 172.21.15.38:0/1554172999 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e8074160 con 0x7fb0e8102ec0 2024-03-26T06:35:33.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e6ffd640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e8101310 0x7fb0e80750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:33.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e67fc640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e8102500 0x7fb0e80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:35:33.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0e77fe640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0e8102ec0 0x7fb0e8073c20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb0e000cad0 tx=0x7fb0e000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:33.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0cffff640 1 -- 172.21.15.38:0/1554172999 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e000ae20 con 0x7fb0e8102ec0 2024-03-26T06:35:33.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0cffff640 1 -- 172.21.15.38:0/1554172999 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0e0005020 con 0x7fb0e8102ec0 2024-03-26T06:35:33.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0ed345640 1 -- 172.21.15.38:0/1554172999 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e8074450 con 0x7fb0e8102ec0 2024-03-26T06:35:33.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0cffff640 1 -- 172.21.15.38:0/1554172999 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e000f6f0 con 0x7fb0e8102ec0 2024-03-26T06:35:33.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.013+0000 7fb0ed345640 1 -- 172.21.15.38:0/1554172999 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e81a60c0 con 0x7fb0e8102ec0 2024-03-26T06:35:33.020 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.014+0000 7fb0ed345640 1 -- 172.21.15.38:0/1554172999 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e81086b0 con 0x7fb0e8102ec0 2024-03-26T06:35:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.015+0000 7fb0cffff640 1 -- 172.21.15.38:0/1554172999 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb0e00051d0 con 0x7fb0e8102ec0 2024-03-26T06:35:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.016+0000 7fb0cffff640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb0d00a2800 0x7fb0d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.016+0000 7fb0e6ffd640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb0d00a2800 0x7fb0d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.016+0000 7fb0cffff640 1 -- 172.21.15.38:0/1554172999 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb0e00cead0 con 0x7fb0e8102ec0 2024-03-26T06:35:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.016+0000 7fb0e6ffd640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb0d00a2800 0x7fb0d00a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb0d802c7b0 tx=0x7fb0d8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:33.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.021+0000 7fb0cffff640 1 -- 172.21.15.38:0/1554172999 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb0e0014030 con 0x7fb0e8102ec0 2024-03-26T06:35:33.145 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b[66818]: 2024-03-26T06:35:32.811+0000 7f0b122e1700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-26T06:35:33.145 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b[66818]: 2024-03-26T06:35:32.811+0000 7f0b122e1700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-26T06:35:33.145 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:35:32 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b[66818]: 2024-03-26T06:35:32.811+0000 7f0b122e1700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-26T06:35:33.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.188+0000 7fb0cffff640 1 -- 172.21.15.38:0/1554172999 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb0e0097b90 con 0x7fb0e8102ec0 2024-03-26T06:35:33.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.238+0000 7fb0ed345640 1 -- 172.21.15.38:0/1554172999 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7fb0e8061730 con 0x7fb0d00a2800 2024-03-26T06:35:33.244 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.242+0000 7fb0cffff640 1 -- 172.21.15.38:0/1554172999 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fb0e8061730 con 0x7fb0d00a2800 2024-03-26T06:35:33.244 INFO:teuthology.orchestra.run.smithi038.stdout:/volumes/qa/sv_0/07e7ee49-7d4c-404e-8471-c5667e5b6b1d 2024-03-26T06:35:33.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 -- 172.21.15.38:0/1554172999 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb0d00a2800 msgr2=0x7fb0d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb0d00a2800 0x7fb0d00a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb0d802c7b0 tx=0x7fb0d8005fd0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 -- 172.21.15.38:0/1554172999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0e8102ec0 msgr2=0x7fb0e8073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0e8102ec0 0x7fb0e8073c20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb0e000cad0 tx=0x7fb0e000cf90 comp rx=0 tx=0).stop 2024-03-26T06:35:33.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 -- 172.21.15.38:0/1554172999 shutdown_connections 2024-03-26T06:35:33.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb0d00a2800 0x7fb0d00a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb0e8102ec0 0x7fb0e8073c20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0e8102500 0x7fb0e80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 --2- 172.21.15.38:0/1554172999 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb0e8101310 0x7fb0e80750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 -- 172.21.15.38:0/1554172999 >> 172.21.15.38:0/1554172999 conn(0x7fb0e80fc8a0 msgr2=0x7fb0e8105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:33.247 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 -- 172.21.15.38:0/1554172999 shutdown_connections 2024-03-26T06:35:33.248 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.244+0000 7fb0ed345640 1 -- 172.21.15.38:0/1554172999 wait complete. 2024-03-26T06:35:33.257 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-26T06:35:33.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.362+0000 7f60780a4640 1 -- 172.21.15.38:0/3685744835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6070106fe0 msgr2=0x7f6070109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.370 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.362+0000 7f60780a4640 1 --2- 172.21.15.38:0/3685744835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6070106fe0 0x7f6070109430 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6060009ca0 tx=0x7f606002c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.362+0000 7f60780a4640 1 -- 172.21.15.38:0/3685744835 shutdown_connections 2024-03-26T06:35:33.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.362+0000 7f60780a4640 1 --2- 172.21.15.38:0/3685744835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6070106fe0 0x7f6070109430 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.362+0000 7f60780a4640 1 --2- 172.21.15.38:0/3685744835 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f60701046b0 0x7f6070106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.362+0000 7f60780a4640 1 --2- 172.21.15.38:0/3685744835 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6070101d80 0x7f6070104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.371 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.362+0000 7f60780a4640 1 -- 172.21.15.38:0/3685744835 >> 172.21.15.38:0/3685744835 conn(0x7f60700fb790 msgr2=0x7f60700fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:33.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.362+0000 7f60780a4640 1 -- 172.21.15.38:0/3685744835 shutdown_connections 2024-03-26T06:35:33.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.362+0000 7f60780a4640 1 -- 172.21.15.38:0/3685744835 wait complete. 2024-03-26T06:35:33.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f60780a4640 1 Processor -- start 2024-03-26T06:35:33.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f60780a4640 1 -- start start 2024-03-26T06:35:33.372 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f60780a4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6070101d80 0x7f6070197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f60780a4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60701046b0 0x7f6070197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f60780a4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6070106fe0 0x7f607019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f60780a4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607010cff0 con 0x7f6070101d80 2024-03-26T06:35:33.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f60780a4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607010ce70 con 0x7f60701046b0 2024-03-26T06:35:33.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f60780a4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607010d170 con 0x7f6070106fe0 2024-03-26T06:35:33.373 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f6075e40640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6070101d80 0x7f6070197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f6075e40640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6070101d80 0x7f6070197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55844/0 (socket says 172.21.15.38:55844) 2024-03-26T06:35:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f6075e40640 1 -- 172.21.15.38:0/2817581480 learned_addr learned my addr 172.21.15.38:0/2817581480 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f607563f640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60701046b0 0x7f6070197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f6075e40640 1 -- 172.21.15.38:0/2817581480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6070106fe0 msgr2=0x7f607019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.374 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f6076641640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6070106fe0 0x7f607019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f6075e40640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6070106fe0 0x7f607019ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f6075e40640 1 -- 172.21.15.38:0/2817581480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60701046b0 msgr2=0x7f6070197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f6075e40640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60701046b0 0x7f6070197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f6075e40640 1 -- 172.21.15.38:0/2817581480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607019f260 con 0x7f6070101d80 2024-03-26T06:35:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.363+0000 7f607563f640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60701046b0 0x7f6070197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:33.375 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.364+0000 7f6075e40640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6070101d80 0x7f6070197640 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f606800d950 tx=0x7f606800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.364+0000 7f606effd640 1 -- 172.21.15.38:0/2817581480 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6068017030 con 0x7f6070101d80 2024-03-26T06:35:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.364+0000 7f606effd640 1 -- 172.21.15.38:0/2817581480 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6068004020 con 0x7f6070101d80 2024-03-26T06:35:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.364+0000 7f60780a4640 1 -- 172.21.15.38:0/2817581480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607019f550 con 0x7f6070101d80 2024-03-26T06:35:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.364+0000 7f606effd640 1 -- 172.21.15.38:0/2817581480 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60680047a0 con 0x7f6070101d80 2024-03-26T06:35:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.364+0000 7f60780a4640 1 -- 172.21.15.38:0/2817581480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607019faa0 con 0x7f6070101d80 2024-03-26T06:35:33.376 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.365+0000 7f60780a4640 1 -- 172.21.15.38:0/2817581480 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60700fd280 con 0x7f6070101d80 2024-03-26T06:35:33.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.366+0000 7f606effd640 1 -- 172.21.15.38:0/2817581480 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f60680041d0 con 0x7f6070101d80 2024-03-26T06:35:33.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.367+0000 7f606effd640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f60580a2890 0x7f60580a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.367+0000 7f607563f640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f60580a2890 0x7f60580a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.377 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.367+0000 7f606effd640 1 -- 172.21.15.38:0/2817581480 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f60680ce540 con 0x7f6070101d80 2024-03-26T06:35:33.378 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.367+0000 7f607563f640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f60580a2890 0x7f60580a4d50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6070198b50 tx=0x7f6064008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:33.379 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.372+0000 7f606effd640 1 -- 172.21.15.38:0/2817581480 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f606809b630 con 0x7f6070101d80 2024-03-26T06:35:33.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:33 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1952208765' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-26T06:35:33.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:33 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] up:active 2024-03-26T06:35:33.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:33 smithi170 ceph-mon[36912]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-26T06:35:33.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:33 smithi170 ceph-mon[36912]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:35:33.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:33 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1952208765' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-26T06:35:33.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:33 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] up:active 2024-03-26T06:35:33.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:33 smithi038 ceph-mon[32265]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-26T06:35:33.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:33 smithi038 ceph-mon[32265]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:35:33.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.586+0000 7f60780a4640 1 -- 172.21.15.38:0/2817581480 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f6070061730 con 0x7f60580a2890 2024-03-26T06:35:33.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.609+0000 7f606effd640 1 -- 172.21.15.38:0/2817581480 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6070061730 con 0x7f60580a2890 2024-03-26T06:35:33.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 -- 172.21.15.38:0/2817581480 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f60580a2890 msgr2=0x7f60580a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f60580a2890 0x7f60580a4d50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6070198b50 tx=0x7f6064008040 comp rx=0 tx=0).stop 2024-03-26T06:35:33.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 -- 172.21.15.38:0/2817581480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6070101d80 msgr2=0x7f6070197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6070101d80 0x7f6070197640 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f606800d950 tx=0x7f606800de10 comp rx=0 tx=0).stop 2024-03-26T06:35:33.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 -- 172.21.15.38:0/2817581480 shutdown_connections 2024-03-26T06:35:33.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f60580a2890 0x7f60580a4d50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6070106fe0 0x7f607019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60701046b0 0x7f6070197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 --2- 172.21.15.38:0/2817581480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6070101d80 0x7f6070197640 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 -- 172.21.15.38:0/2817581480 >> 172.21.15.38:0/2817581480 conn(0x7f60700fb790 msgr2=0x7f6070102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:33.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 -- 172.21.15.38:0/2817581480 shutdown_connections 2024-03-26T06:35:33.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.611+0000 7f60780a4640 1 -- 172.21.15.38:0/2817581480 wait complete. 2024-03-26T06:35:33.625 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-26T06:35:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:33 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1952208765' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-26T06:35:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:33 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] up:active 2024-03-26T06:35:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:33 smithi121 ceph-mon[36349]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-03-26T06:35:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:33 smithi121 ceph-mon[36349]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:35:33.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.730+0000 7f75156dc640 1 -- 172.21.15.38:0/2922945278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7510101e50 msgr2=0x7f7510104240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.730+0000 7f75156dc640 1 --2- 172.21.15.38:0/2922945278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7510101e50 0x7f7510104240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7500009900 tx=0x7f750002bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:33.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.730+0000 7f75156dc640 1 -- 172.21.15.38:0/2922945278 shutdown_connections 2024-03-26T06:35:33.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.730+0000 7f75156dc640 1 --2- 172.21.15.38:0/2922945278 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75101070b0 0x7f75101094a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.730+0000 7f75156dc640 1 --2- 172.21.15.38:0/2922945278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7510104780 0x7f7510106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.730+0000 7f75156dc640 1 --2- 172.21.15.38:0/2922945278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7510101e50 0x7f7510104240 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.730+0000 7f75156dc640 1 -- 172.21.15.38:0/2922945278 >> 172.21.15.38:0/2922945278 conn(0x7f75100fb7c0 msgr2=0x7f75100fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:33.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.730+0000 7f75156dc640 1 -- 172.21.15.38:0/2922945278 shutdown_connections 2024-03-26T06:35:33.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.730+0000 7f75156dc640 1 -- 172.21.15.38:0/2922945278 wait complete. 2024-03-26T06:35:33.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f75156dc640 1 Processor -- start 2024-03-26T06:35:33.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f75156dc640 1 -- start start 2024-03-26T06:35:33.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f75156dc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7510101e50 0x7f7510101770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f75156dc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7510104780 0x7f75100ffe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f75156dc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75101070b0 0x7f7510100340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f75156dc640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751010d000 con 0x7f75101070b0 2024-03-26T06:35:33.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f75156dc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751010ce80 con 0x7f7510104780 2024-03-26T06:35:33.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f75156dc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751010d180 con 0x7f7510101e50 2024-03-26T06:35:33.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f750f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75101070b0 0x7f7510100340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f750f7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75101070b0 0x7f7510100340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55872/0 (socket says 172.21.15.38:55872) 2024-03-26T06:35:33.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f750f7fe640 1 -- 172.21.15.38:0/577035843 learned_addr learned my addr 172.21.15.38:0/577035843 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:33.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f7507fff640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7510104780 0x7f75100ffe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f750effd640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7510101e50 0x7f7510101770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f750f7fe640 1 -- 172.21.15.38:0/577035843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7510101e50 msgr2=0x7f7510101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f750f7fe640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7510101e50 0x7f7510101770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f750f7fe640 1 -- 172.21.15.38:0/577035843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7510104780 msgr2=0x7f75100ffe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f750f7fe640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7510104780 0x7f75100ffe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f750f7fe640 1 -- 172.21.15.38:0/577035843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7510100c00 con 0x7f75101070b0 2024-03-26T06:35:33.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.731+0000 7f750f7fe640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75101070b0 0x7f7510100340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f750800eaa0 tx=0x7f750800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:33.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.732+0000 7f750cff9640 1 -- 172.21.15.38:0/577035843 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750800cdb0 con 0x7f75101070b0 2024-03-26T06:35:33.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.732+0000 7f750cff9640 1 -- 172.21.15.38:0/577035843 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7508005020 con 0x7f75101070b0 2024-03-26T06:35:33.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.732+0000 7f75156dc640 1 -- 172.21.15.38:0/577035843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75101a17c0 con 0x7f75101070b0 2024-03-26T06:35:33.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.732+0000 7f750cff9640 1 -- 172.21.15.38:0/577035843 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75080116b0 con 0x7f75101070b0 2024-03-26T06:35:33.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.732+0000 7f75156dc640 1 -- 172.21.15.38:0/577035843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75101a1d30 con 0x7f75101070b0 2024-03-26T06:35:33.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.733+0000 7f75156dc640 1 -- 172.21.15.38:0/577035843 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75100fd2b0 con 0x7f75101070b0 2024-03-26T06:35:33.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.734+0000 7f750cff9640 1 -- 172.21.15.38:0/577035843 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7508005560 con 0x7f75101070b0 2024-03-26T06:35:33.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.734+0000 7f750cff9640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74fc0a2910 0x7f74fc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:33.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.735+0000 7f750effd640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74fc0a2910 0x7f74fc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:33.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.735+0000 7f750cff9640 1 -- 172.21.15.38:0/577035843 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f75080cec00 con 0x7f75101070b0 2024-03-26T06:35:33.740 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.735+0000 7f750effd640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74fc0a2910 0x7f74fc0a4dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f750002c240 tx=0x7f750002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:33.742 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.739+0000 7f750cff9640 1 -- 172.21.15.38:0/577035843 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f75080d4020 con 0x7f75101070b0 2024-03-26T06:35:33.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.950+0000 7f75156dc640 1 -- 172.21.15.38:0/577035843 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f75100617c0 con 0x7f74fc0a2910 2024-03-26T06:35:33.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.954+0000 7f750cff9640 1 -- 172.21.15.38:0/577035843 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f75100617c0 con 0x7f74fc0a2910 2024-03-26T06:35:33.957 INFO:teuthology.orchestra.run.smithi038.stdout:/volumes/qa/sv_1/a4829ff3-867c-4b07-aa33-9af29bd922f7 2024-03-26T06:35:33.958 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 -- 172.21.15.38:0/577035843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74fc0a2910 msgr2=0x7f74fc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74fc0a2910 0x7f74fc0a4dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f750002c240 tx=0x7f750002df90 comp rx=0 tx=0).stop 2024-03-26T06:35:33.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 -- 172.21.15.38:0/577035843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75101070b0 msgr2=0x7f7510100340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:33.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75101070b0 0x7f7510100340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f750800eaa0 tx=0x7f750800ef60 comp rx=0 tx=0).stop 2024-03-26T06:35:33.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 -- 172.21.15.38:0/577035843 shutdown_connections 2024-03-26T06:35:33.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74fc0a2910 0x7f74fc0a4dd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75101070b0 0x7f7510100340 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7510104780 0x7f75100ffe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 --2- 172.21.15.38:0/577035843 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7510101e50 0x7f7510101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:33.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 -- 172.21.15.38:0/577035843 >> 172.21.15.38:0/577035843 conn(0x7f75100fb7c0 msgr2=0x7f7510102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:33.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 -- 172.21.15.38:0/577035843 shutdown_connections 2024-03-26T06:35:33.959 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:33.956+0000 7f75156dc640 1 -- 172.21.15.38:0/577035843 wait complete. 2024-03-26T06:35:33.970 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:35:34.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.069+0000 7f4c1c80c640 1 -- 172.21.15.38:0/389611712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c181013f0 msgr2=0x7f4c18101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.069+0000 7f4c1c80c640 1 --2- 172.21.15.38:0/389611712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c181013f0 0x7f4c18101870 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4c00009900 tx=0x7f4c0002bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:34.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.069+0000 7f4c1c80c640 1 -- 172.21.15.38:0/389611712 shutdown_connections 2024-03-26T06:35:34.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.069+0000 7f4c1c80c640 1 --2- 172.21.15.38:0/389611712 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c18101db0 0x7f4c18108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.069+0000 7f4c1c80c640 1 --2- 172.21.15.38:0/389611712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c181013f0 0x7f4c18101870 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.069+0000 7f4c1c80c640 1 --2- 172.21.15.38:0/389611712 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c18100200 0x7f4c18100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.069+0000 7f4c1c80c640 1 -- 172.21.15.38:0/389611712 >> 172.21.15.38:0/389611712 conn(0x7f4c180fb770 msgr2=0x7f4c180fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:34.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.069+0000 7f4c1c80c640 1 -- 172.21.15.38:0/389611712 shutdown_connections 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.069+0000 7f4c1c80c640 1 -- 172.21.15.38:0/389611712 wait complete. 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c1c80c640 1 Processor -- start 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c1c80c640 1 -- start start 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c1c80c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c18100200 0x7f4c1806a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c1c80c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c181013f0 0x7f4c1819c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c1c80c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c18101db0 0x7f4c1819e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c1c80c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1810bef0 con 0x7f4c18101db0 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c1c80c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1810bd70 con 0x7f4c18100200 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c1c80c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1810c070 con 0x7f4c181013f0 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c16d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c18101db0 0x7f4c1819e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c16d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c18101db0 0x7f4c1819e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55896/0 (socket says 172.21.15.38:55896) 2024-03-26T06:35:34.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c16d9d640 1 -- 172.21.15.38:0/3051817298 learned_addr learned my addr 172.21.15.38:0/3051817298 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c1659c640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c18100200 0x7f4c1806a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c0ffff640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c181013f0 0x7f4c1819c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c16d9d640 1 -- 172.21.15.38:0/3051817298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c181013f0 msgr2=0x7f4c1819c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c16d9d640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c181013f0 0x7f4c1819c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c16d9d640 1 -- 172.21.15.38:0/3051817298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c18100200 msgr2=0x7f4c1806a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c16d9d640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c18100200 0x7f4c1806a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.070+0000 7f4c16d9d640 1 -- 172.21.15.38:0/3051817298 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1819f290 con 0x7f4c18101db0 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.071+0000 7f4c16d9d640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c18101db0 0x7f4c1819e9d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000eaa0 tx=0x7f4c1000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.071+0000 7f4c0f7fe640 1 -- 172.21.15.38:0/3051817298 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1000cdb0 con 0x7f4c18101db0 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.071+0000 7f4c1c80c640 1 -- 172.21.15.38:0/3051817298 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1819f580 con 0x7f4c18101db0 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.071+0000 7f4c0f7fe640 1 -- 172.21.15.38:0/3051817298 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c10005020 con 0x7f4c18101db0 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.071+0000 7f4c0f7fe640 1 -- 172.21.15.38:0/3051817298 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c100116b0 con 0x7f4c18101db0 2024-03-26T06:35:34.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.071+0000 7f4c1c80c640 1 -- 172.21.15.38:0/3051817298 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c1819fad0 con 0x7f4c18101db0 2024-03-26T06:35:34.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.072+0000 7f4c1c80c640 1 -- 172.21.15.38:0/3051817298 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c181075a0 con 0x7f4c18101db0 2024-03-26T06:35:34.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.073+0000 7f4c0f7fe640 1 -- 172.21.15.38:0/3051817298 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f4c10005560 con 0x7f4c18101db0 2024-03-26T06:35:34.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.073+0000 7f4c0f7fe640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4c040a2890 0x7f4c040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.073+0000 7f4c0f7fe640 1 -- 172.21.15.38:0/3051817298 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4c100ced50 con 0x7f4c18101db0 2024-03-26T06:35:34.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.073+0000 7f4c1659c640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4c040a2890 0x7f4c040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.078 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.074+0000 7f4c1659c640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4c040a2890 0x7f4c040a4d50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4c18101250 tx=0x7f4c08008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:34.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.078+0000 7f4c0f7fe640 1 -- 172.21.15.38:0/3051817298 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4c10016030 con 0x7f4c18101db0 2024-03-26T06:35:34.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.301+0000 7f4c1c80c640 1 -- 172.21.15.38:0/3051817298 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c18061ef0 con 0x7f4c18101db0 2024-03-26T06:35:34.304 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.301+0000 7f4c0f7fe640 1 -- 172.21.15.38:0/3051817298 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11346 (secure 0 0 0) 0x7f4c1009be40 con 0x7f4c18101db0 2024-03-26T06:35:34.304 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:34.304 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/357411663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":357411663},{"type":"v1","addr":"172.21.15.38:6839","nonce":357411663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/654319512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":654319512},{"type":"v1","addr":"172.21.15.38:6841","nonce":654319512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-26T06:35:26.477444+0000","modified":"2024-03-26T06:35:32.167599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24502,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/359437185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":359437185},{"type":"v1","addr":"172.21.15.121:6839","nonce":359437185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-26T06:35:34.304 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 22 2024-03-26T06:35:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 -- 172.21.15.38:0/3051817298 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4c040a2890 msgr2=0x7f4c040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4c040a2890 0x7f4c040a4d50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f4c18101250 tx=0x7f4c08008040 comp rx=0 tx=0).stop 2024-03-26T06:35:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 -- 172.21.15.38:0/3051817298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c18101db0 msgr2=0x7f4c1819e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c18101db0 0x7f4c1819e9d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4c1000eaa0 tx=0x7f4c1000ef60 comp rx=0 tx=0).stop 2024-03-26T06:35:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 -- 172.21.15.38:0/3051817298 shutdown_connections 2024-03-26T06:35:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4c040a2890 0x7f4c040a4d50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c18101db0 0x7f4c1819e9d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4c181013f0 0x7f4c1819c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 --2- 172.21.15.38:0/3051817298 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4c18100200 0x7f4c1806a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 -- 172.21.15.38:0/3051817298 >> 172.21.15.38:0/3051817298 conn(0x7f4c180fb770 msgr2=0x7f4c18104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:34.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 -- 172.21.15.38:0/3051817298 shutdown_connections 2024-03-26T06:35:34.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.303+0000 7f4c1c80c640 1 -- 172.21.15.38:0/3051817298 wait complete. 2024-03-26T06:35:34.315 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:34 smithi038 ceph-mon[32265]: pgmap v289: 97 pgs: 97 active+clean; 580 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s wr, 13 op/s 2024-03-26T06:35:34.316 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:34 smithi038 ceph-mon[32265]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:34.316 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:34 smithi038 ceph-mon[32265]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:34.316 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:34 smithi038 ceph-mon[32265]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-26T06:35:34.317 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:35:34.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.417+0000 7fd4b9220640 1 -- 172.21.15.38:0/827171310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4b4102ec0 msgr2=0x7fd4b4109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.417+0000 7fd4b9220640 1 --2- 172.21.15.38:0/827171310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4b4102ec0 0x7fd4b4109750 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac009ca0 tx=0x7fd4ac02c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.417+0000 7fd4b9220640 1 -- 172.21.15.38:0/827171310 shutdown_connections 2024-03-26T06:35:34.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.417+0000 7fd4b9220640 1 --2- 172.21.15.38:0/827171310 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4b4102ec0 0x7fd4b4109750 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.417+0000 7fd4b9220640 1 --2- 172.21.15.38:0/827171310 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4b4102500 0x7fd4b4102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.417+0000 7fd4b9220640 1 --2- 172.21.15.38:0/827171310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4b4101310 0x7fd4b4101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.417+0000 7fd4b9220640 1 -- 172.21.15.38:0/827171310 >> 172.21.15.38:0/827171310 conn(0x7fd4b40fc8a0 msgr2=0x7fd4b40fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:34.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.417+0000 7fd4b9220640 1 -- 172.21.15.38:0/827171310 shutdown_connections 2024-03-26T06:35:34.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.417+0000 7fd4b9220640 1 -- 172.21.15.38:0/827171310 wait complete. 2024-03-26T06:35:34.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4b9220640 1 Processor -- start 2024-03-26T06:35:34.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4b9220640 1 -- start start 2024-03-26T06:35:34.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4b9220640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4b4101310 0x7fd4b40750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4b9220640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4b4102500 0x7fd4b40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4b9220640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4b4102ec0 0x7fd4b4073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4b9220640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b410d000 con 0x7fd4b4102500 2024-03-26T06:35:34.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4b9220640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b410ce80 con 0x7fd4b4102ec0 2024-03-26T06:35:34.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4b9220640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b410d180 con 0x7fd4b4101310 2024-03-26T06:35:34.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4abfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4b4102500 0x7fd4b40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4abfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4b4102500 0x7fd4b40736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:55906/0 (socket says 172.21.15.38:55906) 2024-03-26T06:35:34.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4abfff640 1 -- 172.21.15.38:0/1437537962 learned_addr learned my addr 172.21.15.38:0/1437537962 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:34.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4b359e640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4b4102ec0 0x7fd4b4073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4b2d9d640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4b4101310 0x7fd4b40750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4abfff640 1 -- 172.21.15.38:0/1437537962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4b4101310 msgr2=0x7fd4b40750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4abfff640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4b4101310 0x7fd4b40750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4abfff640 1 -- 172.21.15.38:0/1437537962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4b4102ec0 msgr2=0x7fd4b4073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4abfff640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4b4102ec0 0x7fd4b4073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4abfff640 1 -- 172.21.15.38:0/1437537962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b40744b0 con 0x7fd4b4102500 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.418+0000 7fd4b359e640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4b4102ec0 0x7fd4b4073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.419+0000 7fd4abfff640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4b4102500 0x7fd4b40736e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c920 tx=0x7fd49c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.419+0000 7fd4b0d99640 1 -- 172.21.15.38:0/1437537962 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c007c70 con 0x7fd4b4102500 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.419+0000 7fd4b9220640 1 -- 172.21.15.38:0/1437537962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b41a16d0 con 0x7fd4b4102500 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.419+0000 7fd4b0d99640 1 -- 172.21.15.38:0/1437537962 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd49c007e10 con 0x7fd4b4102500 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.419+0000 7fd4b0d99640 1 -- 172.21.15.38:0/1437537962 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd49c018890 con 0x7fd4b4102500 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.419+0000 7fd4b9220640 1 -- 172.21.15.38:0/1437537962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b41a1c20 con 0x7fd4b4102500 2024-03-26T06:35:34.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.420+0000 7fd4b9220640 1 -- 172.21.15.38:0/1437537962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b4108560 con 0x7fd4b4102500 2024-03-26T06:35:34.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.421+0000 7fd4b0d99640 1 -- 172.21.15.38:0/1437537962 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd49c018a70 con 0x7fd4b4102500 2024-03-26T06:35:34.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.421+0000 7fd4b0d99640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd4a00a2890 0x7fd4a00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.421+0000 7fd4b2d9d640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd4a00a2890 0x7fd4a00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.421+0000 7fd4b0d99640 1 -- 172.21.15.38:0/1437537962 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd49c0cecd0 con 0x7fd4b4102500 2024-03-26T06:35:34.427 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.422+0000 7fd4b2d9d640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd4a00a2890 0x7fd4a00a4d50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4102360 tx=0x7fd4a4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:34.428 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.426+0000 7fd4b0d99640 1 -- 172.21.15.38:0/1437537962 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd49c014030 con 0x7fd4b4102500 2024-03-26T06:35:34.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.609+0000 7fd4b9220640 1 -- 172.21.15.38:0/1437537962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4b402d0c0 con 0x7fd4b4102500 2024-03-26T06:35:34.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.610+0000 7fd4b0d99640 1 -- 172.21.15.38:0/1437537962 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22209 (secure 0 0 0) 0x7fd49c01f300 con 0x7fd4b4102500 2024-03-26T06:35:34.624 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:34.624 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":86,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:35:28.120088+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:35:34.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 -- 172.21.15.38:0/1437537962 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd4a00a2890 msgr2=0x7fd4a00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd4a00a2890 0x7fd4a00a4d50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd4b4102360 tx=0x7fd4a4008040 comp rx=0 tx=0).stop 2024-03-26T06:35:34.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 -- 172.21.15.38:0/1437537962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4b4102500 msgr2=0x7fd4b40736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4b4102500 0x7fd4b40736e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd49c00c920 tx=0x7fd49c00cde0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 -- 172.21.15.38:0/1437537962 shutdown_connections 2024-03-26T06:35:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd4a00a2890 0x7fd4a00a4d50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4b4102ec0 0x7fd4b4073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd4b4102500 0x7fd4b40736e0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 --2- 172.21.15.38:0/1437537962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd4b4101310 0x7fd4b40750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 -- 172.21.15.38:0/1437537962 >> 172.21.15.38:0/1437537962 conn(0x7fd4b40fc8a0 msgr2=0x7fd4b4105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 -- 172.21.15.38:0/1437537962 shutdown_connections 2024-03-26T06:35:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.612+0000 7fd4b9220640 1 -- 172.21.15.38:0/1437537962 wait complete. 2024-03-26T06:35:34.627 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-26T06:35:34.627 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-26T06:35:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:34 smithi121 ceph-mon[36349]: pgmap v289: 97 pgs: 97 active+clean; 580 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s wr, 13 op/s 2024-03-26T06:35:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:34 smithi121 ceph-mon[36349]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:34 smithi121 ceph-mon[36349]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:34 smithi121 ceph-mon[36349]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-26T06:35:34.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:34 smithi170 ceph-mon[36912]: pgmap v289: 97 pgs: 97 active+clean; 580 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s wr, 13 op/s 2024-03-26T06:35:34.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:34 smithi170 ceph-mon[36912]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:34.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:34 smithi170 ceph-mon[36912]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:34.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:34 smithi170 ceph-mon[36912]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-26T06:35:34.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.728+0000 7ff4b4468640 1 -- 172.21.15.38:0/2340439936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ac101420 msgr2=0x7ff4ac1018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.728+0000 7ff4b4468640 1 --2- 172.21.15.38:0/2340439936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ac101420 0x7ff4ac1018a0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009900 tx=0x7ff49c02bc30 comp rx=0 tx=0).stop 2024-03-26T06:35:34.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.728+0000 7ff4b4468640 1 -- 172.21.15.38:0/2340439936 shutdown_connections 2024-03-26T06:35:34.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.728+0000 7ff4b4468640 1 --2- 172.21.15.38:0/2340439936 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ac101de0 0x7ff4ac108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.728+0000 7ff4b4468640 1 --2- 172.21.15.38:0/2340439936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ac101420 0x7ff4ac1018a0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.728+0000 7ff4b4468640 1 --2- 172.21.15.38:0/2340439936 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4ac100230 0x7ff4ac100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.728+0000 7ff4b4468640 1 -- 172.21.15.38:0/2340439936 >> 172.21.15.38:0/2340439936 conn(0x7ff4ac0fb7a0 msgr2=0x7ff4ac0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:34.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.728+0000 7ff4b4468640 1 -- 172.21.15.38:0/2340439936 shutdown_connections 2024-03-26T06:35:34.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.728+0000 7ff4b4468640 1 -- 172.21.15.38:0/2340439936 wait complete. 2024-03-26T06:35:34.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.728+0000 7ff4b4468640 1 Processor -- start 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.728+0000 7ff4b4468640 1 -- start start 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b4468640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4ac100230 0x7ff4ac1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b4468640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ac101420 0x7ff4ac199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b4468640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ac101de0 0x7ff4ac1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b4468640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac10bec0 con 0x7ff4ac101420 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b4468640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac10bd40 con 0x7ff4ac101de0 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b4468640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4ac10c040 con 0x7ff4ac100230 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b1a03640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ac101420 0x7ff4ac199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b1a03640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ac101420 0x7ff4ac199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50078/0 (socket says 172.21.15.38:50078) 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b1a03640 1 -- 172.21.15.38:0/3500754173 learned_addr learned my addr 172.21.15.38:0/3500754173 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b2a05640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ac101de0 0x7ff4ac1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b2204640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4ac100230 0x7ff4ac1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b1a03640 1 -- 172.21.15.38:0/3500754173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4ac100230 msgr2=0x7ff4ac1997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b1a03640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4ac100230 0x7ff4ac1997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b1a03640 1 -- 172.21.15.38:0/3500754173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ac101de0 msgr2=0x7ff4ac1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b1a03640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ac101de0 0x7ff4ac1a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b1a03640 1 -- 172.21.15.38:0/3500754173 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac1a1410 con 0x7ff4ac101420 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b2a05640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ac101de0 0x7ff4ac1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b1a03640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ac101420 0x7ff4ac199d30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009230 tx=0x7ff49c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4a37fe640 1 -- 172.21.15.38:0/3500754173 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c041070 con 0x7ff4ac101420 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4a37fe640 1 -- 172.21.15.38:0/3500754173 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff49c0054e0 con 0x7ff4ac101420 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b4468640 1 -- 172.21.15.38:0/3500754173 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4ac1a16a0 con 0x7ff4ac101420 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4a37fe640 1 -- 172.21.15.38:0/3500754173 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c034440 con 0x7ff4ac101420 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.729+0000 7ff4b4468640 1 -- 172.21.15.38:0/3500754173 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4ac1a1b90 con 0x7ff4ac101420 2024-03-26T06:35:34.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.730+0000 7ff4b4468640 1 -- 172.21.15.38:0/3500754173 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4ac1a1df0 con 0x7ff4ac101420 2024-03-26T06:35:34.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.731+0000 7ff4a37fe640 1 -- 172.21.15.38:0/3500754173 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff49c044070 con 0x7ff4ac101420 2024-03-26T06:35:34.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.732+0000 7ff4a37fe640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff4980a2890 0x7ff4980a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:34.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.732+0000 7ff4b2204640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff4980a2890 0x7ff4980a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:34.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.732+0000 7ff4a37fe640 1 -- 172.21.15.38:0/3500754173 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff49c0f07f0 con 0x7ff4ac101420 2024-03-26T06:35:34.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.733+0000 7ff4b2204640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff4980a2890 0x7ff4980a4d50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4005fd0 tx=0x7ff4a4005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:34.739 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.737+0000 7ff4a37fe640 1 -- 172.21.15.38:0/3500754173 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff49c0bd8f0 con 0x7ff4ac101420 2024-03-26T06:35:34.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.916+0000 7ff4b4468640 1 -- 172.21.15.38:0/3500754173 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff4ac06a030 con 0x7ff4980a2890 2024-03-26T06:35:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.923+0000 7ff4a37fe640 1 -- 172.21.15.38:0/3500754173 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258709 (secure 0 0 0) 0x7ff4ac06a030 con 0x7ff4980a2890 2024-03-26T06:35:34.929 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:34.931 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 -- 172.21.15.38:0/3500754173 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff4980a2890 msgr2=0x7ff4980a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff4980a2890 0x7ff4980a4d50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4005fd0 tx=0x7ff4a4005850 comp rx=0 tx=0).stop 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 -- 172.21.15.38:0/3500754173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ac101420 msgr2=0x7ff4ac199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ac101420 0x7ff4ac199d30 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009230 tx=0x7ff49c005dd0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 -- 172.21.15.38:0/3500754173 shutdown_connections 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff4980a2890 0x7ff4980a4d50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4ac101de0 0x7ff4ac1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4ac101420 0x7ff4ac199d30 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 --2- 172.21.15.38:0/3500754173 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4ac100230 0x7ff4ac1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 -- 172.21.15.38:0/3500754173 >> 172.21.15.38:0/3500754173 conn(0x7ff4ac0fb7a0 msgr2=0x7ff4ac069920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 -- 172.21.15.38:0/3500754173 shutdown_connections 2024-03-26T06:35:34.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:34.926+0000 7ff4b4468640 1 -- 172.21.15.38:0/3500754173 wait complete. 2024-03-26T06:35:34.940 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-03-26T06:35:34.234705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593580,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":261,"num_write_kb":2962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":244,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370216,"kb_used_data":12704,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571608,"statfs":{"total":1151940427776,"available":1151561326592,"internally_reserved":0,"allocated":13008896,"data_stored":7080934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3212,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001677"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388036+0000","last_change":"2024-03-26T06:35:27.137165+0000","last_active":"2024-03-26T06:35:28.388036+0000","last_peered":"2024-03-26T06:35:28.388036+0000","last_clean":"2024-03-26T06:35:28.388036+0000","last_became_active":"2024-03-26T06:35:27.136869+0000","last_became_peered":"2024-03-26T06:35:27.136869+0000","last_unstale":"2024-03-26T06:35:28.388036+0000","last_undegraded":"2024-03-26T06:35:28.388036+0000","last_fullsized":"2024-03-26T06:35:28.388036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:43:36.671967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387117+0000","last_change":"2024-03-26T06:35:27.130619+0000","last_active":"2024-03-26T06:35:28.387117+0000","last_peered":"2024-03-26T06:35:28.387117+0000","last_clean":"2024-03-26T06:35:28.387117+0000","last_became_active":"2024-03-26T06:35:27.128846+0000","last_became_peered":"2024-03-26T06:35:27.128846+0000","last_unstale":"2024-03-26T06:35:28.387117+0000","last_undegraded":"2024-03-26T06:35:28.387117+0000","last_fullsized":"2024-03-26T06:35:28.387117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:14:15.627268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387152+0000","last_change":"2024-03-26T06:35:27.138214+0000","last_active":"2024-03-26T06:35:28.387152+0000","last_peered":"2024-03-26T06:35:28.387152+0000","last_clean":"2024-03-26T06:35:28.387152+0000","last_became_active":"2024-03-26T06:35:27.137846+0000","last_became_peered":"2024-03-26T06:35:27.137846+0000","last_unstale":"2024-03-26T06:35:28.387152+0000","last_undegraded":"2024-03-26T06:35:28.387152+0000","last_fullsized":"2024-03-26T06:35:28.387152+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:48:46.441889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124881+0000","last_change":"2024-03-26T06:35:27.129582+0000","last_active":"2024-03-26T06:35:28.124881+0000","last_peered":"2024-03-26T06:35:28.124881+0000","last_clean":"2024-03-26T06:35:28.124881+0000","last_became_active":"2024-03-26T06:35:27.129245+0000","last_became_peered":"2024-03-26T06:35:27.129245+0000","last_unstale":"2024-03-26T06:35:28.124881+0000","last_undegraded":"2024-03-26T06:35:28.124881+0000","last_fullsized":"2024-03-26T06:35:28.124881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:09:14.548067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315409+0000","last_change":"2024-03-26T06:35:27.131993+0000","last_active":"2024-03-26T06:35:28.315409+0000","last_peered":"2024-03-26T06:35:28.315409+0000","last_clean":"2024-03-26T06:35:28.315409+0000","last_became_active":"2024-03-26T06:35:27.131650+0000","last_became_peered":"2024-03-26T06:35:27.131650+0000","last_unstale":"2024-03-26T06:35:28.315409+0000","last_undegraded":"2024-03-26T06:35:28.315409+0000","last_fullsized":"2024-03-26T06:35:28.315409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:38:41.407314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388036+0000","last_change":"2024-03-26T06:35:27.130325+0000","last_active":"2024-03-26T06:35:28.388036+0000","last_peered":"2024-03-26T06:35:28.388036+0000","last_clean":"2024-03-26T06:35:28.388036+0000","last_became_active":"2024-03-26T06:35:27.129658+0000","last_became_peered":"2024-03-26T06:35:27.129658+0000","last_unstale":"2024-03-26T06:35:28.388036+0000","last_undegraded":"2024-03-26T06:35:28.388036+0000","last_fullsized":"2024-03-26T06:35:28.388036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:25:18.580839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387686+0000","last_change":"2024-03-26T06:35:27.130870+0000","last_active":"2024-03-26T06:35:28.387686+0000","last_peered":"2024-03-26T06:35:28.387686+0000","last_clean":"2024-03-26T06:35:28.387686+0000","last_became_active":"2024-03-26T06:35:27.130521+0000","last_became_peered":"2024-03-26T06:35:27.130521+0000","last_unstale":"2024-03-26T06:35:28.387686+0000","last_undegraded":"2024-03-26T06:35:28.387686+0000","last_fullsized":"2024-03-26T06:35:28.387686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:58:40.640708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125807+0000","last_change":"2024-03-26T06:35:27.136258+0000","last_active":"2024-03-26T06:35:28.125807+0000","last_peered":"2024-03-26T06:35:28.125807+0000","last_clean":"2024-03-26T06:35:28.125807+0000","last_became_active":"2024-03-26T06:35:27.135746+0000","last_became_peered":"2024-03-26T06:35:27.135746+0000","last_unstale":"2024-03-26T06:35:28.125807+0000","last_undegraded":"2024-03-26T06:35:28.125807+0000","last_fullsized":"2024-03-26T06:35:28.125807+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:42:24.270717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388694+0000","last_change":"2024-03-26T06:35:27.130533+0000","last_active":"2024-03-26T06:35:28.388694+0000","last_peered":"2024-03-26T06:35:28.388694+0000","last_clean":"2024-03-26T06:35:28.388694+0000","last_became_active":"2024-03-26T06:35:27.130251+0000","last_became_peered":"2024-03-26T06:35:27.130251+0000","last_unstale":"2024-03-26T06:35:28.388694+0000","last_undegraded":"2024-03-26T06:35:28.388694+0000","last_fullsized":"2024-03-26T06:35:28.388694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T18:13:40.209389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388817+0000","last_change":"2024-03-26T06:35:27.130494+0000","last_active":"2024-03-26T06:35:28.388817+0000","last_peered":"2024-03-26T06:35:28.388817+0000","last_clean":"2024-03-26T06:35:28.388817+0000","last_became_active":"2024-03-26T06:35:27.130048+0000","last_became_peered":"2024-03-26T06:35:27.130048+0000","last_unstale":"2024-03-26T06:35:28.388817+0000","last_undegraded":"2024-03-26T06:35:28.388817+0000","last_fullsized":"2024-03-26T06:35:28.388817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:34:19.649515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388336+0000","last_change":"2024-03-26T06:35:27.137285+0000","last_active":"2024-03-26T06:35:28.388336+0000","last_peered":"2024-03-26T06:35:28.388336+0000","last_clean":"2024-03-26T06:35:28.388336+0000","last_became_active":"2024-03-26T06:35:27.136930+0000","last_became_peered":"2024-03-26T06:35:27.136930+0000","last_unstale":"2024-03-26T06:35:28.388336+0000","last_undegraded":"2024-03-26T06:35:28.388336+0000","last_fullsized":"2024-03-26T06:35:28.388336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:36:42.036591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124923+0000","last_change":"2024-03-26T06:35:27.133279+0000","last_active":"2024-03-26T06:35:28.124923+0000","last_peered":"2024-03-26T06:35:28.124923+0000","last_clean":"2024-03-26T06:35:28.124923+0000","last_became_active":"2024-03-26T06:35:27.132860+0000","last_became_peered":"2024-03-26T06:35:27.132860+0000","last_unstale":"2024-03-26T06:35:28.124923+0000","last_undegraded":"2024-03-26T06:35:28.124923+0000","last_fullsized":"2024-03-26T06:35:28.124923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:36:34.455411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387010+0000","last_change":"2024-03-26T06:35:27.124143+0000","last_active":"2024-03-26T06:35:28.387010+0000","last_peered":"2024-03-26T06:35:28.387010+0000","last_clean":"2024-03-26T06:35:28.387010+0000","last_became_active":"2024-03-26T06:35:27.123774+0000","last_became_peered":"2024-03-26T06:35:27.123774+0000","last_unstale":"2024-03-26T06:35:28.387010+0000","last_undegraded":"2024-03-26T06:35:28.387010+0000","last_fullsized":"2024-03-26T06:35:28.387010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:05:14.135531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387457+0000","last_change":"2024-03-26T06:35:27.127523+0000","last_active":"2024-03-26T06:35:28.387457+0000","last_peered":"2024-03-26T06:35:28.387457+0000","last_clean":"2024-03-26T06:35:28.387457+0000","last_became_active":"2024-03-26T06:35:27.127141+0000","last_became_peered":"2024-03-26T06:35:27.127141+0000","last_unstale":"2024-03-26T06:35:28.387457+0000","last_undegraded":"2024-03-26T06:35:28.387457+0000","last_fullsized":"2024-03-26T06:35:28.387457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:21:44.524721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.386959+0000","last_change":"2024-03-26T06:35:27.131744+0000","last_active":"2024-03-26T06:35:28.386959+0000","last_peered":"2024-03-26T06:35:28.386959+0000","last_clean":"2024-03-26T06:35:28.386959+0000","last_became_active":"2024-03-26T06:35:27.128735+0000","last_became_peered":"2024-03-26T06:35:27.128735+0000","last_unstale":"2024-03-26T06:35:28.386959+0000","last_undegraded":"2024-03-26T06:35:28.386959+0000","last_fullsized":"2024-03-26T06:35:28.386959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:08:18.457258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.386999+0000","last_change":"2024-03-26T06:35:27.132021+0000","last_active":"2024-03-26T06:35:28.386999+0000","last_peered":"2024-03-26T06:35:28.386999+0000","last_clean":"2024-03-26T06:35:28.386999+0000","last_became_active":"2024-03-26T06:35:27.131722+0000","last_became_peered":"2024-03-26T06:35:27.131722+0000","last_unstale":"2024-03-26T06:35:28.386999+0000","last_undegraded":"2024-03-26T06:35:28.386999+0000","last_fullsized":"2024-03-26T06:35:28.386999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:52:23.253250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388227+0000","last_change":"2024-03-26T06:35:27.137225+0000","last_active":"2024-03-26T06:35:28.388227+0000","last_peered":"2024-03-26T06:35:28.388227+0000","last_clean":"2024-03-26T06:35:28.388227+0000","last_became_active":"2024-03-26T06:35:27.136935+0000","last_became_peered":"2024-03-26T06:35:27.136935+0000","last_unstale":"2024-03-26T06:35:28.388227+0000","last_undegraded":"2024-03-26T06:35:28.388227+0000","last_fullsized":"2024-03-26T06:35:28.388227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:14:02.540482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.313039+0000","last_change":"2024-03-26T06:35:27.136068+0000","last_active":"2024-03-26T06:35:28.313039+0000","last_peered":"2024-03-26T06:35:28.313039+0000","last_clean":"2024-03-26T06:35:28.313039+0000","last_became_active":"2024-03-26T06:35:27.135586+0000","last_became_peered":"2024-03-26T06:35:27.135586+0000","last_unstale":"2024-03-26T06:35:28.313039+0000","last_undegraded":"2024-03-26T06:35:28.313039+0000","last_fullsized":"2024-03-26T06:35:28.313039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:21:51.729675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387550+0000","last_change":"2024-03-26T06:35:27.131576+0000","last_active":"2024-03-26T06:35:28.387550+0000","last_peered":"2024-03-26T06:35:28.387550+0000","last_clean":"2024-03-26T06:35:28.387550+0000","last_became_active":"2024-03-26T06:35:27.131262+0000","last_became_peered":"2024-03-26T06:35:27.131262+0000","last_unstale":"2024-03-26T06:35:28.387550+0000","last_undegraded":"2024-03-26T06:35:28.387550+0000","last_fullsized":"2024-03-26T06:35:28.387550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:50:13.688512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:31.170968+0000","last_change":"2024-03-26T06:35:26.106217+0000","last_active":"2024-03-26T06:35:31.170968+0000","last_peered":"2024-03-26T06:35:31.170968+0000","last_clean":"2024-03-26T06:35:31.170968+0000","last_became_active":"2024-03-26T06:35:26.105866+0000","last_became_peered":"2024-03-26T06:35:26.105866+0000","last_unstale":"2024-03-26T06:35:31.170968+0000","last_undegraded":"2024-03-26T06:35:31.170968+0000","last_fullsized":"2024-03-26T06:35:31.170968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:52:04.770844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126319+0000","last_change":"2024-03-26T06:35:27.141510+0000","last_active":"2024-03-26T06:35:28.126319+0000","last_peered":"2024-03-26T06:35:28.126319+0000","last_clean":"2024-03-26T06:35:28.126319+0000","last_became_active":"2024-03-26T06:35:27.141182+0000","last_became_peered":"2024-03-26T06:35:27.141182+0000","last_unstale":"2024-03-26T06:35:28.126319+0000","last_undegraded":"2024-03-26T06:35:28.126319+0000","last_fullsized":"2024-03-26T06:35:28.126319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:40:55.993939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126585+0000","last_change":"2024-03-26T06:35:26.095502+0000","last_active":"2024-03-26T06:35:28.126585+0000","last_peered":"2024-03-26T06:35:28.126585+0000","last_clean":"2024-03-26T06:35:28.126585+0000","last_became_active":"2024-03-26T06:35:26.095168+0000","last_became_peered":"2024-03-26T06:35:26.095168+0000","last_unstale":"2024-03-26T06:35:28.126585+0000","last_undegraded":"2024-03-26T06:35:28.126585+0000","last_fullsized":"2024-03-26T06:35:28.126585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:01:44.307274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387572+0000","last_change":"2024-03-26T06:35:27.137179+0000","last_active":"2024-03-26T06:35:28.387572+0000","last_peered":"2024-03-26T06:35:28.387572+0000","last_clean":"2024-03-26T06:35:28.387572+0000","last_became_active":"2024-03-26T06:35:27.136660+0000","last_became_peered":"2024-03-26T06:35:27.136660+0000","last_unstale":"2024-03-26T06:35:28.387572+0000","last_undegraded":"2024-03-26T06:35:28.387572+0000","last_fullsized":"2024-03-26T06:35:28.387572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:16:16.042134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315749+0000","last_change":"2024-03-26T06:35:26.105395+0000","last_active":"2024-03-26T06:35:28.315749+0000","last_peered":"2024-03-26T06:35:28.315749+0000","last_clean":"2024-03-26T06:35:28.315749+0000","last_became_active":"2024-03-26T06:35:26.104879+0000","last_became_peered":"2024-03-26T06:35:26.104879+0000","last_unstale":"2024-03-26T06:35:28.315749+0000","last_undegraded":"2024-03-26T06:35:28.315749+0000","last_fullsized":"2024-03-26T06:35:28.315749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:44:57.444844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387205+0000","last_change":"2024-03-26T06:35:27.133549+0000","last_active":"2024-03-26T06:35:28.387205+0000","last_peered":"2024-03-26T06:35:28.387205+0000","last_clean":"2024-03-26T06:35:28.387205+0000","last_became_active":"2024-03-26T06:35:27.132176+0000","last_became_peered":"2024-03-26T06:35:27.132176+0000","last_unstale":"2024-03-26T06:35:28.387205+0000","last_undegraded":"2024-03-26T06:35:28.387205+0000","last_fullsized":"2024-03-26T06:35:28.387205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:15:59.281268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315597+0000","last_change":"2024-03-26T06:35:26.101438+0000","last_active":"2024-03-26T06:35:28.315597+0000","last_peered":"2024-03-26T06:35:28.315597+0000","last_clean":"2024-03-26T06:35:28.315597+0000","last_became_active":"2024-03-26T06:35:26.101139+0000","last_became_peered":"2024-03-26T06:35:26.101139+0000","last_unstale":"2024-03-26T06:35:28.315597+0000","last_undegraded":"2024-03-26T06:35:28.315597+0000","last_fullsized":"2024-03-26T06:35:28.315597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:10:36.750046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125198+0000","last_change":"2024-03-26T06:35:27.127896+0000","last_active":"2024-03-26T06:35:28.125198+0000","last_peered":"2024-03-26T06:35:28.125198+0000","last_clean":"2024-03-26T06:35:28.125198+0000","last_became_active":"2024-03-26T06:35:27.127558+0000","last_became_peered":"2024-03-26T06:35:27.127558+0000","last_unstale":"2024-03-26T06:35:28.125198+0000","last_undegraded":"2024-03-26T06:35:28.125198+0000","last_fullsized":"2024-03-26T06:35:28.125198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:04:58.033255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124716+0000","last_change":"2024-03-26T06:35:26.100148+0000","last_active":"2024-03-26T06:35:28.124716+0000","last_peered":"2024-03-26T06:35:28.124716+0000","last_clean":"2024-03-26T06:35:28.124716+0000","last_became_active":"2024-03-26T06:35:26.099827+0000","last_became_peered":"2024-03-26T06:35:26.099827+0000","last_unstale":"2024-03-26T06:35:28.124716+0000","last_undegraded":"2024-03-26T06:35:28.124716+0000","last_fullsized":"2024-03-26T06:35:28.124716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:33:21.845866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315457+0000","last_change":"2024-03-26T06:35:27.125318+0000","last_active":"2024-03-26T06:35:28.315457+0000","last_peered":"2024-03-26T06:35:28.315457+0000","last_clean":"2024-03-26T06:35:28.315457+0000","last_became_active":"2024-03-26T06:35:27.125013+0000","last_became_peered":"2024-03-26T06:35:27.125013+0000","last_unstale":"2024-03-26T06:35:28.315457+0000","last_undegraded":"2024-03-26T06:35:28.315457+0000","last_fullsized":"2024-03-26T06:35:28.315457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:50:25.123317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387754+0000","last_change":"2024-03-26T06:35:26.097150+0000","last_active":"2024-03-26T06:35:28.387754+0000","last_peered":"2024-03-26T06:35:28.387754+0000","last_clean":"2024-03-26T06:35:28.387754+0000","last_became_active":"2024-03-26T06:35:26.096844+0000","last_became_peered":"2024-03-26T06:35:26.096844+0000","last_unstale":"2024-03-26T06:35:28.387754+0000","last_undegraded":"2024-03-26T06:35:28.387754+0000","last_fullsized":"2024-03-26T06:35:28.387754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:57:47.399493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124891+0000","last_change":"2024-03-26T06:35:27.132261+0000","last_active":"2024-03-26T06:35:28.124891+0000","last_peered":"2024-03-26T06:35:28.124891+0000","last_clean":"2024-03-26T06:35:28.124891+0000","last_became_active":"2024-03-26T06:35:27.131893+0000","last_became_peered":"2024-03-26T06:35:27.131893+0000","last_unstale":"2024-03-26T06:35:28.124891+0000","last_undegraded":"2024-03-26T06:35:28.124891+0000","last_fullsized":"2024-03-26T06:35:28.124891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:42.521515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.386661+0000","last_change":"2024-03-26T06:35:26.092356+0000","last_active":"2024-03-26T06:35:28.386661+0000","last_peered":"2024-03-26T06:35:28.386661+0000","last_clean":"2024-03-26T06:35:28.386661+0000","last_became_active":"2024-03-26T06:35:26.091948+0000","last_became_peered":"2024-03-26T06:35:26.091948+0000","last_unstale":"2024-03-26T06:35:28.386661+0000","last_undegraded":"2024-03-26T06:35:28.386661+0000","last_fullsized":"2024-03-26T06:35:28.386661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:48:47.011884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125923+0000","last_change":"2024-03-26T06:35:27.129880+0000","last_active":"2024-03-26T06:35:28.125923+0000","last_peered":"2024-03-26T06:35:28.125923+0000","last_clean":"2024-03-26T06:35:28.125923+0000","last_became_active":"2024-03-26T06:35:27.129513+0000","last_became_peered":"2024-03-26T06:35:27.129513+0000","last_unstale":"2024-03-26T06:35:28.125923+0000","last_undegraded":"2024-03-26T06:35:28.125923+0000","last_fullsized":"2024-03-26T06:35:28.125923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:10:53.747657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124504+0000","last_change":"2024-03-26T06:35:26.095603+0000","last_active":"2024-03-26T06:35:28.124504+0000","last_peered":"2024-03-26T06:35:28.124504+0000","last_clean":"2024-03-26T06:35:28.124504+0000","last_became_active":"2024-03-26T06:35:26.095309+0000","last_became_peered":"2024-03-26T06:35:26.095309+0000","last_unstale":"2024-03-26T06:35:28.124504+0000","last_undegraded":"2024-03-26T06:35:28.124504+0000","last_fullsized":"2024-03-26T06:35:28.124504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:30:55.373598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.312889+0000","last_change":"2024-03-26T06:35:27.133271+0000","last_active":"2024-03-26T06:35:28.312889+0000","last_peered":"2024-03-26T06:35:28.312889+0000","last_clean":"2024-03-26T06:35:28.312889+0000","last_became_active":"2024-03-26T06:35:27.132553+0000","last_became_peered":"2024-03-26T06:35:27.132553+0000","last_unstale":"2024-03-26T06:35:28.312889+0000","last_undegraded":"2024-03-26T06:35:28.312889+0000","last_fullsized":"2024-03-26T06:35:28.312889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:13:59.021131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.158550+0000","last_change":"2024-03-26T06:35:26.094323+0000","last_active":"2024-03-26T06:35:30.158550+0000","last_peered":"2024-03-26T06:35:30.158550+0000","last_clean":"2024-03-26T06:35:30.158550+0000","last_became_active":"2024-03-26T06:35:26.093814+0000","last_became_peered":"2024-03-26T06:35:26.093814+0000","last_unstale":"2024-03-26T06:35:30.158550+0000","last_undegraded":"2024-03-26T06:35:30.158550+0000","last_fullsized":"2024-03-26T06:35:30.158550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:11.450427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388143+0000","last_change":"2024-03-26T06:35:27.127398+0000","last_active":"2024-03-26T06:35:28.388143+0000","last_peered":"2024-03-26T06:35:28.388143+0000","last_clean":"2024-03-26T06:35:28.388143+0000","last_became_active":"2024-03-26T06:35:27.127054+0000","last_became_peered":"2024-03-26T06:35:27.127054+0000","last_unstale":"2024-03-26T06:35:28.388143+0000","last_undegraded":"2024-03-26T06:35:28.388143+0000","last_fullsized":"2024-03-26T06:35:28.388143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:51:04.150954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387996+0000","last_change":"2024-03-26T06:35:26.098531+0000","last_active":"2024-03-26T06:35:28.387996+0000","last_peered":"2024-03-26T06:35:28.387996+0000","last_clean":"2024-03-26T06:35:28.387996+0000","last_became_active":"2024-03-26T06:35:26.097942+0000","last_became_peered":"2024-03-26T06:35:26.097942+0000","last_unstale":"2024-03-26T06:35:28.387996+0000","last_undegraded":"2024-03-26T06:35:28.387996+0000","last_fullsized":"2024-03-26T06:35:28.387996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:44:59.884627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387508+0000","last_change":"2024-03-26T06:35:27.136145+0000","last_active":"2024-03-26T06:35:28.387508+0000","last_peered":"2024-03-26T06:35:28.387508+0000","last_clean":"2024-03-26T06:35:28.387508+0000","last_became_active":"2024-03-26T06:35:27.135560+0000","last_became_peered":"2024-03-26T06:35:27.135560+0000","last_unstale":"2024-03-26T06:35:28.387508+0000","last_undegraded":"2024-03-26T06:35:28.387508+0000","last_fullsized":"2024-03-26T06:35:28.387508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:19:01.991175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125211+0000","last_change":"2024-03-26T06:35:26.095119+0000","last_active":"2024-03-26T06:35:28.125211+0000","last_peered":"2024-03-26T06:35:28.125211+0000","last_clean":"2024-03-26T06:35:28.125211+0000","last_became_active":"2024-03-26T06:35:26.094789+0000","last_became_peered":"2024-03-26T06:35:26.094789+0000","last_unstale":"2024-03-26T06:35:28.125211+0000","last_undegraded":"2024-03-26T06:35:28.125211+0000","last_fullsized":"2024-03-26T06:35:28.125211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:59:07.626207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387410+0000","last_change":"2024-03-26T06:35:27.133722+0000","last_active":"2024-03-26T06:35:28.387410+0000","last_peered":"2024-03-26T06:35:28.387410+0000","last_clean":"2024-03-26T06:35:28.387410+0000","last_became_active":"2024-03-26T06:35:27.132313+0000","last_became_peered":"2024-03-26T06:35:27.132313+0000","last_unstale":"2024-03-26T06:35:28.387410+0000","last_undegraded":"2024-03-26T06:35:28.387410+0000","last_fullsized":"2024-03-26T06:35:28.387410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:58:48.350708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.158968+0000","last_change":"2024-03-26T06:35:26.095229+0000","last_active":"2024-03-26T06:35:30.158968+0000","last_peered":"2024-03-26T06:35:30.158968+0000","last_clean":"2024-03-26T06:35:30.158968+0000","last_became_active":"2024-03-26T06:35:26.094929+0000","last_became_peered":"2024-03-26T06:35:26.094929+0000","last_unstale":"2024-03-26T06:35:30.158968+0000","last_undegraded":"2024-03-26T06:35:30.158968+0000","last_fullsized":"2024-03-26T06:35:30.158968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:49:33.080958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315596+0000","last_change":"2024-03-26T06:35:27.127735+0000","last_active":"2024-03-26T06:35:28.315596+0000","last_peered":"2024-03-26T06:35:28.315596+0000","last_clean":"2024-03-26T06:35:28.315596+0000","last_became_active":"2024-03-26T06:35:27.127434+0000","last_became_peered":"2024-03-26T06:35:27.127434+0000","last_unstale":"2024-03-26T06:35:28.315596+0000","last_undegraded":"2024-03-26T06:35:28.315596+0000","last_fullsized":"2024-03-26T06:35:28.315596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:30:09.745415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.158641+0000","last_change":"2024-03-26T06:35:26.095943+0000","last_active":"2024-03-26T06:35:30.158641+0000","last_peered":"2024-03-26T06:35:30.158641+0000","last_clean":"2024-03-26T06:35:30.158641+0000","last_became_active":"2024-03-26T06:35:26.095577+0000","last_became_peered":"2024-03-26T06:35:26.095577+0000","last_unstale":"2024-03-26T06:35:30.158641+0000","last_undegraded":"2024-03-26T06:35:30.158641+0000","last_fullsized":"2024-03-26T06:35:30.158641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:42:34.459051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.386780+0000","last_change":"2024-03-26T06:35:27.133199+0000","last_active":"2024-03-26T06:35:28.386780+0000","last_peered":"2024-03-26T06:35:28.386780+0000","last_clean":"2024-03-26T06:35:28.386780+0000","last_became_active":"2024-03-26T06:35:27.132187+0000","last_became_peered":"2024-03-26T06:35:27.132187+0000","last_unstale":"2024-03-26T06:35:28.386780+0000","last_undegraded":"2024-03-26T06:35:28.386780+0000","last_fullsized":"2024-03-26T06:35:28.386780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:16:22.832804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124833+0000","last_change":"2024-03-26T06:35:26.100902+0000","last_active":"2024-03-26T06:35:28.124833+0000","last_peered":"2024-03-26T06:35:28.124833+0000","last_clean":"2024-03-26T06:35:28.124833+0000","last_became_active":"2024-03-26T06:35:26.100509+0000","last_became_peered":"2024-03-26T06:35:26.100509+0000","last_unstale":"2024-03-26T06:35:28.124833+0000","last_undegraded":"2024-03-26T06:35:28.124833+0000","last_fullsized":"2024-03-26T06:35:28.124833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:52:53.459610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388014+0000","last_change":"2024-03-26T06:35:27.129126+0000","last_active":"2024-03-26T06:35:28.388014+0000","last_peered":"2024-03-26T06:35:28.388014+0000","last_clean":"2024-03-26T06:35:28.388014+0000","last_became_active":"2024-03-26T06:35:27.128741+0000","last_became_peered":"2024-03-26T06:35:27.128741+0000","last_unstale":"2024-03-26T06:35:28.388014+0000","last_undegraded":"2024-03-26T06:35:28.388014+0000","last_fullsized":"2024-03-26T06:35:28.388014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:09:15.613369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:31.168340+0000","last_change":"2024-03-26T06:35:26.096239+0000","last_active":"2024-03-26T06:35:31.168340+0000","last_peered":"2024-03-26T06:35:31.168340+0000","last_clean":"2024-03-26T06:35:31.168340+0000","last_became_active":"2024-03-26T06:35:26.095915+0000","last_became_peered":"2024-03-26T06:35:26.095915+0000","last_unstale":"2024-03-26T06:35:31.168340+0000","last_undegraded":"2024-03-26T06:35:31.168340+0000","last_fullsized":"2024-03-26T06:35:31.168340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:50:24.566322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388526+0000","last_change":"2024-03-26T06:35:27.125846+0000","last_active":"2024-03-26T06:35:28.388526+0000","last_peered":"2024-03-26T06:35:28.388526+0000","last_clean":"2024-03-26T06:35:28.388526+0000","last_became_active":"2024-03-26T06:35:27.125515+0000","last_became_peered":"2024-03-26T06:35:27.125515+0000","last_unstale":"2024-03-26T06:35:28.388526+0000","last_undegraded":"2024-03-26T06:35:28.388526+0000","last_fullsized":"2024-03-26T06:35:28.388526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:02:17.358507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315156+0000","last_change":"2024-03-26T06:35:26.102857+0000","last_active":"2024-03-26T06:35:28.315156+0000","last_peered":"2024-03-26T06:35:28.315156+0000","last_clean":"2024-03-26T06:35:28.315156+0000","last_became_active":"2024-03-26T06:35:26.102552+0000","last_became_peered":"2024-03-26T06:35:26.102552+0000","last_unstale":"2024-03-26T06:35:28.315156+0000","last_undegraded":"2024-03-26T06:35:28.315156+0000","last_fullsized":"2024-03-26T06:35:28.315156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:31:30.833012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125177+0000","last_change":"2024-03-26T06:35:27.133496+0000","last_active":"2024-03-26T06:35:28.125177+0000","last_peered":"2024-03-26T06:35:28.125177+0000","last_clean":"2024-03-26T06:35:28.125177+0000","last_became_active":"2024-03-26T06:35:27.133162+0000","last_became_peered":"2024-03-26T06:35:27.133162+0000","last_unstale":"2024-03-26T06:35:28.125177+0000","last_undegraded":"2024-03-26T06:35:28.125177+0000","last_fullsized":"2024-03-26T06:35:28.125177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:00:48.927846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315675+0000","last_change":"2024-03-26T06:35:26.103876+0000","last_active":"2024-03-26T06:35:28.315675+0000","last_peered":"2024-03-26T06:35:28.315675+0000","last_clean":"2024-03-26T06:35:28.315675+0000","last_became_active":"2024-03-26T06:35:26.102981+0000","last_became_peered":"2024-03-26T06:35:26.102981+0000","last_unstale":"2024-03-26T06:35:28.315675+0000","last_undegraded":"2024-03-26T06:35:28.315675+0000","last_fullsized":"2024-03-26T06:35:28.315675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:24:04.154212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387576+0000","last_change":"2024-03-26T06:35:27.127529+0000","last_active":"2024-03-26T06:35:28.387576+0000","last_peered":"2024-03-26T06:35:28.387576+0000","last_clean":"2024-03-26T06:35:28.387576+0000","last_became_active":"2024-03-26T06:35:27.127208+0000","last_became_peered":"2024-03-26T06:35:27.127208+0000","last_unstale":"2024-03-26T06:35:28.387576+0000","last_undegraded":"2024-03-26T06:35:28.387576+0000","last_fullsized":"2024-03-26T06:35:28.387576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:29:35.313042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.312949+0000","last_change":"2024-03-26T06:35:26.099309+0000","last_active":"2024-03-26T06:35:28.312949+0000","last_peered":"2024-03-26T06:35:28.312949+0000","last_clean":"2024-03-26T06:35:28.312949+0000","last_became_active":"2024-03-26T06:35:26.098948+0000","last_became_peered":"2024-03-26T06:35:26.098948+0000","last_unstale":"2024-03-26T06:35:28.312949+0000","last_undegraded":"2024-03-26T06:35:28.312949+0000","last_fullsized":"2024-03-26T06:35:28.312949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:49:35.210018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388763+0000","last_change":"2024-03-26T06:35:27.132441+0000","last_active":"2024-03-26T06:35:28.388763+0000","last_peered":"2024-03-26T06:35:28.388763+0000","last_clean":"2024-03-26T06:35:28.388763+0000","last_became_active":"2024-03-26T06:35:27.131846+0000","last_became_peered":"2024-03-26T06:35:27.131846+0000","last_unstale":"2024-03-26T06:35:28.388763+0000","last_undegraded":"2024-03-26T06:35:28.388763+0000","last_fullsized":"2024-03-26T06:35:28.388763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:14.779299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387970+0000","last_change":"2024-03-26T06:35:26.107214+0000","last_active":"2024-03-26T06:35:28.387970+0000","last_peered":"2024-03-26T06:35:28.387970+0000","last_clean":"2024-03-26T06:35:28.387970+0000","last_became_active":"2024-03-26T06:35:26.106935+0000","last_became_peered":"2024-03-26T06:35:26.106935+0000","last_unstale":"2024-03-26T06:35:28.387970+0000","last_undegraded":"2024-03-26T06:35:28.387970+0000","last_fullsized":"2024-03-26T06:35:28.387970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:49:33.101354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388280+0000","last_change":"2024-03-26T06:35:27.131374+0000","last_active":"2024-03-26T06:35:28.388280+0000","last_peered":"2024-03-26T06:35:28.388280+0000","last_clean":"2024-03-26T06:35:28.388280+0000","last_became_active":"2024-03-26T06:35:27.131030+0000","last_became_peered":"2024-03-26T06:35:27.131030+0000","last_unstale":"2024-03-26T06:35:28.388280+0000","last_undegraded":"2024-03-26T06:35:28.388280+0000","last_fullsized":"2024-03-26T06:35:28.388280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:33:40.809278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124706+0000","last_change":"2024-03-26T06:35:27.127240+0000","last_active":"2024-03-26T06:35:28.124706+0000","last_peered":"2024-03-26T06:35:28.124706+0000","last_clean":"2024-03-26T06:35:28.124706+0000","last_became_active":"2024-03-26T06:35:27.126923+0000","last_became_peered":"2024-03-26T06:35:27.126923+0000","last_unstale":"2024-03-26T06:35:28.124706+0000","last_undegraded":"2024-03-26T06:35:28.124706+0000","last_fullsized":"2024-03-26T06:35:28.124706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:28:20.422536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.386604+0000","last_change":"2024-03-26T06:35:26.096718+0000","last_active":"2024-03-26T06:35:28.386604+0000","last_peered":"2024-03-26T06:35:28.386604+0000","last_clean":"2024-03-26T06:35:28.386604+0000","last_became_active":"2024-03-26T06:35:26.096407+0000","last_became_peered":"2024-03-26T06:35:26.096407+0000","last_unstale":"2024-03-26T06:35:28.386604+0000","last_undegraded":"2024-03-26T06:35:28.386604+0000","last_fullsized":"2024-03-26T06:35:28.386604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:12:00.799567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126374+0000","last_change":"2024-03-26T06:35:27.138485+0000","last_active":"2024-03-26T06:35:28.126374+0000","last_peered":"2024-03-26T06:35:28.126374+0000","last_clean":"2024-03-26T06:35:28.126374+0000","last_became_active":"2024-03-26T06:35:27.138114+0000","last_became_peered":"2024-03-26T06:35:27.138114+0000","last_unstale":"2024-03-26T06:35:28.126374+0000","last_undegraded":"2024-03-26T06:35:28.126374+0000","last_fullsized":"2024-03-26T06:35:28.126374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:28:25.244683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.157548+0000","last_change":"2024-03-26T06:35:26.096111+0000","last_active":"2024-03-26T06:35:30.157548+0000","last_peered":"2024-03-26T06:35:30.157548+0000","last_clean":"2024-03-26T06:35:30.157548+0000","last_became_active":"2024-03-26T06:35:26.095809+0000","last_became_peered":"2024-03-26T06:35:26.095809+0000","last_unstale":"2024-03-26T06:35:30.157548+0000","last_undegraded":"2024-03-26T06:35:30.157548+0000","last_fullsized":"2024-03-26T06:35:30.157548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:04:17.436961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":348,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388229+0000","last_change":"2024-03-26T06:34:13.886034+0000","last_active":"2024-03-26T06:35:28.388229+0000","last_peered":"2024-03-26T06:35:28.388229+0000","last_clean":"2024-03-26T06:35:28.388229+0000","last_became_active":"2024-03-26T06:34:13.885472+0000","last_became_peered":"2024-03-26T06:34:13.885472+0000","last_unstale":"2024-03-26T06:35:28.388229+0000","last_undegraded":"2024-03-26T06:35:28.388229+0000","last_fullsized":"2024-03-26T06:35:28.388229+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:30:52.854577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:30:52.854577+0000","last_clean_scrub_stamp":"2024-03-26T06:30:52.854577+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:43:35.315474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387189+0000","last_change":"2024-03-26T06:35:27.131778+0000","last_active":"2024-03-26T06:35:28.387189+0000","last_peered":"2024-03-26T06:35:28.387189+0000","last_clean":"2024-03-26T06:35:28.387189+0000","last_became_active":"2024-03-26T06:35:27.131433+0000","last_became_peered":"2024-03-26T06:35:27.131433+0000","last_unstale":"2024-03-26T06:35:28.387189+0000","last_undegraded":"2024-03-26T06:35:28.387189+0000","last_fullsized":"2024-03-26T06:35:28.387189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:17:58.614371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388726+0000","last_change":"2024-03-26T06:35:26.106052+0000","last_active":"2024-03-26T06:35:28.388726+0000","last_peered":"2024-03-26T06:35:28.388726+0000","last_clean":"2024-03-26T06:35:28.388726+0000","last_became_active":"2024-03-26T06:35:26.105728+0000","last_became_peered":"2024-03-26T06:35:26.105728+0000","last_unstale":"2024-03-26T06:35:28.388726+0000","last_undegraded":"2024-03-26T06:35:28.388726+0000","last_fullsized":"2024-03-26T06:35:28.388726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:19:12.226026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315564+0000","last_change":"2024-03-26T06:35:26.099487+0000","last_active":"2024-03-26T06:35:28.315564+0000","last_peered":"2024-03-26T06:35:28.315564+0000","last_clean":"2024-03-26T06:35:28.315564+0000","last_became_active":"2024-03-26T06:35:26.099180+0000","last_became_peered":"2024-03-26T06:35:26.099180+0000","last_unstale":"2024-03-26T06:35:28.315564+0000","last_undegraded":"2024-03-26T06:35:28.315564+0000","last_fullsized":"2024-03-26T06:35:28.315564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:15:20.832451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387142+0000","last_change":"2024-03-26T06:35:27.125799+0000","last_active":"2024-03-26T06:35:28.387142+0000","last_peered":"2024-03-26T06:35:28.387142+0000","last_clean":"2024-03-26T06:35:28.387142+0000","last_became_active":"2024-03-26T06:35:27.125323+0000","last_became_peered":"2024-03-26T06:35:27.125323+0000","last_unstale":"2024-03-26T06:35:28.387142+0000","last_undegraded":"2024-03-26T06:35:28.387142+0000","last_fullsized":"2024-03-26T06:35:28.387142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:07:31.162007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:31.170858+0000","last_change":"2024-03-26T06:35:26.103348+0000","last_active":"2024-03-26T06:35:31.170858+0000","last_peered":"2024-03-26T06:35:31.170858+0000","last_clean":"2024-03-26T06:35:31.170858+0000","last_became_active":"2024-03-26T06:35:26.103025+0000","last_became_peered":"2024-03-26T06:35:26.103025+0000","last_unstale":"2024-03-26T06:35:31.170858+0000","last_undegraded":"2024-03-26T06:35:31.170858+0000","last_fullsized":"2024-03-26T06:35:31.170858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:51:51.581237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126431+0000","last_change":"2024-03-26T06:35:27.128853+0000","last_active":"2024-03-26T06:35:28.126431+0000","last_peered":"2024-03-26T06:35:28.126431+0000","last_clean":"2024-03-26T06:35:28.126431+0000","last_became_active":"2024-03-26T06:35:27.128472+0000","last_became_peered":"2024-03-26T06:35:27.128472+0000","last_unstale":"2024-03-26T06:35:28.126431+0000","last_undegraded":"2024-03-26T06:35:28.126431+0000","last_fullsized":"2024-03-26T06:35:28.126431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:01:01.574623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.157678+0000","last_change":"2024-03-26T06:35:26.100523+0000","last_active":"2024-03-26T06:35:30.157678+0000","last_peered":"2024-03-26T06:35:30.157678+0000","last_clean":"2024-03-26T06:35:30.157678+0000","last_became_active":"2024-03-26T06:35:26.099313+0000","last_became_peered":"2024-03-26T06:35:26.099313+0000","last_unstale":"2024-03-26T06:35:30.157678+0000","last_undegraded":"2024-03-26T06:35:30.157678+0000","last_fullsized":"2024-03-26T06:35:30.157678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:43:44.182464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387380+0000","last_change":"2024-03-26T06:35:27.131438+0000","last_active":"2024-03-26T06:35:28.387380+0000","last_peered":"2024-03-26T06:35:28.387380+0000","last_clean":"2024-03-26T06:35:28.387380+0000","last_became_active":"2024-03-26T06:35:27.131112+0000","last_became_peered":"2024-03-26T06:35:27.131112+0000","last_unstale":"2024-03-26T06:35:28.387380+0000","last_undegraded":"2024-03-26T06:35:28.387380+0000","last_fullsized":"2024-03-26T06:35:28.387380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:07.621191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.157557+0000","last_change":"2024-03-26T06:35:26.105039+0000","last_active":"2024-03-26T06:35:30.157557+0000","last_peered":"2024-03-26T06:35:30.157557+0000","last_clean":"2024-03-26T06:35:30.157557+0000","last_became_active":"2024-03-26T06:35:26.104776+0000","last_became_peered":"2024-03-26T06:35:26.104776+0000","last_unstale":"2024-03-26T06:35:30.157557+0000","last_undegraded":"2024-03-26T06:35:30.157557+0000","last_fullsized":"2024-03-26T06:35:30.157557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:20:17.748144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315283+0000","last_change":"2024-03-26T06:35:27.126320+0000","last_active":"2024-03-26T06:35:28.315283+0000","last_peered":"2024-03-26T06:35:28.315283+0000","last_clean":"2024-03-26T06:35:28.315283+0000","last_became_active":"2024-03-26T06:35:27.125996+0000","last_became_peered":"2024-03-26T06:35:27.125996+0000","last_unstale":"2024-03-26T06:35:28.315283+0000","last_undegraded":"2024-03-26T06:35:28.315283+0000","last_fullsized":"2024-03-26T06:35:28.315283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:08:21.386787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.158403+0000","last_change":"2024-03-26T06:35:26.103303+0000","last_active":"2024-03-26T06:35:30.158403+0000","last_peered":"2024-03-26T06:35:30.158403+0000","last_clean":"2024-03-26T06:35:30.158403+0000","last_became_active":"2024-03-26T06:35:26.102311+0000","last_became_peered":"2024-03-26T06:35:26.102311+0000","last_unstale":"2024-03-26T06:35:30.158403+0000","last_undegraded":"2024-03-26T06:35:30.158403+0000","last_fullsized":"2024-03-26T06:35:30.158403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:44:13.845386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387347+0000","last_change":"2024-03-26T06:35:27.128845+0000","last_active":"2024-03-26T06:35:28.387347+0000","last_peered":"2024-03-26T06:35:28.387347+0000","last_clean":"2024-03-26T06:35:28.387347+0000","last_became_active":"2024-03-26T06:35:27.128532+0000","last_became_peered":"2024-03-26T06:35:27.128532+0000","last_unstale":"2024-03-26T06:35:28.387347+0000","last_undegraded":"2024-03-26T06:35:28.387347+0000","last_fullsized":"2024-03-26T06:35:28.387347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:35:08.087689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387442+0000","last_change":"2024-03-26T06:35:26.107589+0000","last_active":"2024-03-26T06:35:28.387442+0000","last_peered":"2024-03-26T06:35:28.387442+0000","last_clean":"2024-03-26T06:35:28.387442+0000","last_became_active":"2024-03-26T06:35:26.107280+0000","last_became_peered":"2024-03-26T06:35:26.107280+0000","last_unstale":"2024-03-26T06:35:28.387442+0000","last_undegraded":"2024-03-26T06:35:28.387442+0000","last_fullsized":"2024-03-26T06:35:28.387442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:57:27.621484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126155+0000","last_change":"2024-03-26T06:35:27.131238+0000","last_active":"2024-03-26T06:35:28.126155+0000","last_peered":"2024-03-26T06:35:28.126155+0000","last_clean":"2024-03-26T06:35:28.126155+0000","last_became_active":"2024-03-26T06:35:27.130902+0000","last_became_peered":"2024-03-26T06:35:27.130902+0000","last_unstale":"2024-03-26T06:35:28.126155+0000","last_undegraded":"2024-03-26T06:35:28.126155+0000","last_fullsized":"2024-03-26T06:35:28.126155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:02:47.180199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315271+0000","last_change":"2024-03-26T06:35:26.105886+0000","last_active":"2024-03-26T06:35:28.315271+0000","last_peered":"2024-03-26T06:35:28.315271+0000","last_clean":"2024-03-26T06:35:28.315271+0000","last_became_active":"2024-03-26T06:35:26.105429+0000","last_became_peered":"2024-03-26T06:35:26.105429+0000","last_unstale":"2024-03-26T06:35:28.315271+0000","last_undegraded":"2024-03-26T06:35:28.315271+0000","last_fullsized":"2024-03-26T06:35:28.315271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:51:57.897588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388160+0000","last_change":"2024-03-26T06:35:27.133273+0000","last_active":"2024-03-26T06:35:28.388160+0000","last_peered":"2024-03-26T06:35:28.388160+0000","last_clean":"2024-03-26T06:35:28.388160+0000","last_became_active":"2024-03-26T06:35:27.132955+0000","last_became_peered":"2024-03-26T06:35:27.132955+0000","last_unstale":"2024-03-26T06:35:28.388160+0000","last_undegraded":"2024-03-26T06:35:28.388160+0000","last_fullsized":"2024-03-26T06:35:28.388160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:26:01.542473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387802+0000","last_change":"2024-03-26T06:35:26.103065+0000","last_active":"2024-03-26T06:35:28.387802+0000","last_peered":"2024-03-26T06:35:28.387802+0000","last_clean":"2024-03-26T06:35:28.387802+0000","last_became_active":"2024-03-26T06:35:26.102761+0000","last_became_peered":"2024-03-26T06:35:26.102761+0000","last_unstale":"2024-03-26T06:35:28.387802+0000","last_undegraded":"2024-03-26T06:35:28.387802+0000","last_fullsized":"2024-03-26T06:35:28.387802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:09:27.650223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388509+0000","last_change":"2024-03-26T06:35:27.137820+0000","last_active":"2024-03-26T06:35:28.388509+0000","last_peered":"2024-03-26T06:35:28.388509+0000","last_clean":"2024-03-26T06:35:28.388509+0000","last_became_active":"2024-03-26T06:35:27.137375+0000","last_became_peered":"2024-03-26T06:35:27.137375+0000","last_unstale":"2024-03-26T06:35:28.388509+0000","last_undegraded":"2024-03-26T06:35:28.388509+0000","last_fullsized":"2024-03-26T06:35:28.388509+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:43:13.908662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124984+0000","last_change":"2024-03-26T06:35:26.094295+0000","last_active":"2024-03-26T06:35:28.124984+0000","last_peered":"2024-03-26T06:35:28.124984+0000","last_clean":"2024-03-26T06:35:28.124984+0000","last_became_active":"2024-03-26T06:35:26.093950+0000","last_became_peered":"2024-03-26T06:35:26.093950+0000","last_unstale":"2024-03-26T06:35:28.124984+0000","last_undegraded":"2024-03-26T06:35:28.124984+0000","last_fullsized":"2024-03-26T06:35:28.124984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:45:23.700237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388195+0000","last_change":"2024-03-26T06:35:27.131253+0000","last_active":"2024-03-26T06:35:28.388195+0000","last_peered":"2024-03-26T06:35:28.388195+0000","last_clean":"2024-03-26T06:35:28.388195+0000","last_became_active":"2024-03-26T06:35:27.130929+0000","last_became_peered":"2024-03-26T06:35:27.130929+0000","last_unstale":"2024-03-26T06:35:28.388195+0000","last_undegraded":"2024-03-26T06:35:28.388195+0000","last_fullsized":"2024-03-26T06:35:28.388195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:45:55.744139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387106+0000","last_change":"2024-03-26T06:35:26.104272+0000","last_active":"2024-03-26T06:35:28.387106+0000","last_peered":"2024-03-26T06:35:28.387106+0000","last_clean":"2024-03-26T06:35:28.387106+0000","last_became_active":"2024-03-26T06:35:26.103957+0000","last_became_peered":"2024-03-26T06:35:26.103957+0000","last_unstale":"2024-03-26T06:35:28.387106+0000","last_undegraded":"2024-03-26T06:35:28.387106+0000","last_fullsized":"2024-03-26T06:35:28.387106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:09:29.037044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388594+0000","last_change":"2024-03-26T06:35:27.128221+0000","last_active":"2024-03-26T06:35:28.388594+0000","last_peered":"2024-03-26T06:35:28.388594+0000","last_clean":"2024-03-26T06:35:28.388594+0000","last_became_active":"2024-03-26T06:35:27.126609+0000","last_became_peered":"2024-03-26T06:35:27.126609+0000","last_unstale":"2024-03-26T06:35:28.388594+0000","last_undegraded":"2024-03-26T06:35:28.388594+0000","last_fullsized":"2024-03-26T06:35:28.388594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:59:56.314853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387544+0000","last_change":"2024-03-26T06:35:27.122330+0000","last_active":"2024-03-26T06:35:28.387544+0000","last_peered":"2024-03-26T06:35:28.387544+0000","last_clean":"2024-03-26T06:35:28.387544+0000","last_became_active":"2024-03-26T06:35:27.121989+0000","last_became_peered":"2024-03-26T06:35:27.121989+0000","last_unstale":"2024-03-26T06:35:28.387544+0000","last_undegraded":"2024-03-26T06:35:28.387544+0000","last_fullsized":"2024-03-26T06:35:28.387544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:51:29.611555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387710+0000","last_change":"2024-03-26T06:35:27.127975+0000","last_active":"2024-03-26T06:35:28.387710+0000","last_peered":"2024-03-26T06:35:28.387710+0000","last_clean":"2024-03-26T06:35:28.387710+0000","last_became_active":"2024-03-26T06:35:27.127635+0000","last_became_peered":"2024-03-26T06:35:27.127635+0000","last_unstale":"2024-03-26T06:35:28.387710+0000","last_undegraded":"2024-03-26T06:35:28.387710+0000","last_fullsized":"2024-03-26T06:35:28.387710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:48.587117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387732+0000","last_change":"2024-03-26T06:35:27.133538+0000","last_active":"2024-03-26T06:35:28.387732+0000","last_peered":"2024-03-26T06:35:28.387732+0000","last_clean":"2024-03-26T06:35:28.387732+0000","last_became_active":"2024-03-26T06:35:27.133104+0000","last_became_peered":"2024-03-26T06:35:27.133104+0000","last_unstale":"2024-03-26T06:35:28.387732+0000","last_undegraded":"2024-03-26T06:35:28.387732+0000","last_fullsized":"2024-03-26T06:35:28.387732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:20:10.417567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387515+0000","last_change":"2024-03-26T06:35:27.129503+0000","last_active":"2024-03-26T06:35:28.387515+0000","last_peered":"2024-03-26T06:35:28.387515+0000","last_clean":"2024-03-26T06:35:28.387515+0000","last_became_active":"2024-03-26T06:35:27.128930+0000","last_became_peered":"2024-03-26T06:35:27.128930+0000","last_unstale":"2024-03-26T06:35:28.387515+0000","last_undegraded":"2024-03-26T06:35:28.387515+0000","last_fullsized":"2024-03-26T06:35:28.387515+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:23:37.476593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387061+0000","last_change":"2024-03-26T06:35:27.137385+0000","last_active":"2024-03-26T06:35:28.387061+0000","last_peered":"2024-03-26T06:35:28.387061+0000","last_clean":"2024-03-26T06:35:28.387061+0000","last_became_active":"2024-03-26T06:35:27.136969+0000","last_became_peered":"2024-03-26T06:35:27.136969+0000","last_unstale":"2024-03-26T06:35:28.387061+0000","last_undegraded":"2024-03-26T06:35:28.387061+0000","last_fullsized":"2024-03-26T06:35:28.387061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:40:16.722381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387451+0000","last_change":"2024-03-26T06:35:27.125287+0000","last_active":"2024-03-26T06:35:28.387451+0000","last_peered":"2024-03-26T06:35:28.387451+0000","last_clean":"2024-03-26T06:35:28.387451+0000","last_became_active":"2024-03-26T06:35:27.124274+0000","last_became_peered":"2024-03-26T06:35:27.124274+0000","last_unstale":"2024-03-26T06:35:28.387451+0000","last_undegraded":"2024-03-26T06:35:28.387451+0000","last_fullsized":"2024-03-26T06:35:28.387451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:33:43.461949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387965+0000","last_change":"2024-03-26T06:35:27.124805+0000","last_active":"2024-03-26T06:35:28.387965+0000","last_peered":"2024-03-26T06:35:28.387965+0000","last_clean":"2024-03-26T06:35:28.387965+0000","last_became_active":"2024-03-26T06:35:27.123947+0000","last_became_peered":"2024-03-26T06:35:27.123947+0000","last_unstale":"2024-03-26T06:35:28.387965+0000","last_undegraded":"2024-03-26T06:35:28.387965+0000","last_fullsized":"2024-03-26T06:35:28.387965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:47.273083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126289+0000","last_change":"2024-03-26T06:35:27.141167+0000","last_active":"2024-03-26T06:35:28.126289+0000","last_peered":"2024-03-26T06:35:28.126289+0000","last_clean":"2024-03-26T06:35:28.126289+0000","last_became_active":"2024-03-26T06:35:27.140781+0000","last_became_peered":"2024-03-26T06:35:27.140781+0000","last_unstale":"2024-03-26T06:35:28.126289+0000","last_undegraded":"2024-03-26T06:35:28.126289+0000","last_fullsized":"2024-03-26T06:35:28.126289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:42:36.705671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124686+0000","last_change":"2024-03-26T06:35:27.132336+0000","last_active":"2024-03-26T06:35:28.124686+0000","last_peered":"2024-03-26T06:35:28.124686+0000","last_clean":"2024-03-26T06:35:28.124686+0000","last_became_active":"2024-03-26T06:35:27.131944+0000","last_became_peered":"2024-03-26T06:35:27.131944+0000","last_unstale":"2024-03-26T06:35:28.124686+0000","last_undegraded":"2024-03-26T06:35:28.124686+0000","last_fullsized":"2024-03-26T06:35:28.124686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:51:57.916873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125876+0000","last_change":"2024-03-26T06:35:27.141297+0000","last_active":"2024-03-26T06:35:28.125876+0000","last_peered":"2024-03-26T06:35:28.125876+0000","last_clean":"2024-03-26T06:35:28.125876+0000","last_became_active":"2024-03-26T06:35:27.140932+0000","last_became_peered":"2024-03-26T06:35:27.140932+0000","last_unstale":"2024-03-26T06:35:28.125876+0000","last_undegraded":"2024-03-26T06:35:28.125876+0000","last_fullsized":"2024-03-26T06:35:28.125876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:17.942885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387211+0000","last_change":"2024-03-26T06:35:27.135381+0000","last_active":"2024-03-26T06:35:28.387211+0000","last_peered":"2024-03-26T06:35:28.387211+0000","last_clean":"2024-03-26T06:35:28.387211+0000","last_became_active":"2024-03-26T06:35:27.135053+0000","last_became_peered":"2024-03-26T06:35:27.135053+0000","last_unstale":"2024-03-26T06:35:28.387211+0000","last_undegraded":"2024-03-26T06:35:28.387211+0000","last_fullsized":"2024-03-26T06:35:28.387211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:40:21.960182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387033+0000","last_change":"2024-03-26T06:35:27.130780+0000","last_active":"2024-03-26T06:35:28.387033+0000","last_peered":"2024-03-26T06:35:28.387033+0000","last_clean":"2024-03-26T06:35:28.387033+0000","last_became_active":"2024-03-26T06:35:27.130171+0000","last_became_peered":"2024-03-26T06:35:27.130171+0000","last_unstale":"2024-03-26T06:35:28.387033+0000","last_undegraded":"2024-03-26T06:35:28.387033+0000","last_fullsized":"2024-03-26T06:35:28.387033+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:53:18.053617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387779+0000","last_change":"2024-03-26T06:35:27.129677+0000","last_active":"2024-03-26T06:35:28.387779+0000","last_peered":"2024-03-26T06:35:28.387779+0000","last_clean":"2024-03-26T06:35:28.387779+0000","last_became_active":"2024-03-26T06:35:27.129275+0000","last_became_peered":"2024-03-26T06:35:27.129275+0000","last_unstale":"2024-03-26T06:35:28.387779+0000","last_undegraded":"2024-03-26T06:35:28.387779+0000","last_fullsized":"2024-03-26T06:35:28.387779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:18.851198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3212,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":11622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":442154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28532,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716620,"statfs":{"total":95995035648,"available":95965818880,"internally_reserved":0,"allocated":1552384,"data_stored":1034508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776151,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":442356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":929792,"data_stored":442176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299425,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32672,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712480,"statfs":{"total":95995035648,"available":95961579520,"internally_reserved":0,"allocated":1519616,"data_stored":1032522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462951,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":441572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":443184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":443050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40424,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704728,"statfs":{"total":95995035648,"available":95953641472,"internally_reserved":0,"allocated":946176,"data_stored":443840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":441856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888113,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":441594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1527808,"data_stored":1032122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T06:35:34.947 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-26T06:35:35.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.080+0000 7ff089dd1640 1 -- 172.21.15.38:0/3287291265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff084101e40 msgr2=0x7ff0841086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:35.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.080+0000 7ff089dd1640 1 --2- 172.21.15.38:0/3287291265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff084101e40 0x7ff0841086d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009ca0 tx=0x7ff07c02c0f0 comp rx=0 tx=0).stop 2024-03-26T06:35:35.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.080+0000 7ff089dd1640 1 -- 172.21.15.38:0/3287291265 shutdown_connections 2024-03-26T06:35:35.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.080+0000 7ff089dd1640 1 --2- 172.21.15.38:0/3287291265 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff084101e40 0x7ff0841086d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:35.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.080+0000 7ff089dd1640 1 --2- 172.21.15.38:0/3287291265 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff084101480 0x7ff084101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:35.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.080+0000 7ff089dd1640 1 --2- 172.21.15.38:0/3287291265 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff084100290 0x7ff0841006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:35.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.080+0000 7ff089dd1640 1 -- 172.21.15.38:0/3287291265 >> 172.21.15.38:0/3287291265 conn(0x7ff0840fb820 msgr2=0x7ff0840fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:35.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.080+0000 7ff089dd1640 1 -- 172.21.15.38:0/3287291265 shutdown_connections 2024-03-26T06:35:35.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.080+0000 7ff089dd1640 1 -- 172.21.15.38:0/3287291265 wait complete. 2024-03-26T06:35:35.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.080+0000 7ff089dd1640 1 Processor -- start 2024-03-26T06:35:35.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff089dd1640 1 -- start start 2024-03-26T06:35:35.085 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff089dd1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff084100290 0x7ff0841975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff089dd1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff084101480 0x7ff084197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff089dd1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff084101e40 0x7ff08419eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff089dd1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08410beb0 con 0x7ff084101e40 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff089dd1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08410bd30 con 0x7ff084101480 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff089dd1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08410c030 con 0x7ff084100290 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff083fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff084101e40 0x7ff08419eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff083fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff084101e40 0x7ff08419eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50100/0 (socket says 172.21.15.38:50100) 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff083fff640 1 -- 172.21.15.38:0/2994452880 learned_addr learned my addr 172.21.15.38:0/2994452880 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff082ffd640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff084101480 0x7ff084197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff0837fe640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff084100290 0x7ff0841975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff083fff640 1 -- 172.21.15.38:0/2994452880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff084100290 msgr2=0x7ff0841975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:35.086 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff083fff640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff084100290 0x7ff0841975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff083fff640 1 -- 172.21.15.38:0/2994452880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff084101480 msgr2=0x7ff084197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff083fff640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff084101480 0x7ff084197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff083fff640 1 -- 172.21.15.38:0/2994452880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08419f1c0 con 0x7ff084101e40 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff0837fe640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff084100290 0x7ff0841975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff082ffd640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff084101480 0x7ff084197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff083fff640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff084101e40 0x7ff08419eb50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff07c02c700 tx=0x7ff07c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.081+0000 7ff080ff9640 1 -- 172.21.15.38:0/2994452880 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c039040 con 0x7ff084101e40 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.082+0000 7ff080ff9640 1 -- 172.21.15.38:0/2994452880 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff07c044070 con 0x7ff084101e40 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.082+0000 7ff089dd1640 1 -- 172.21.15.38:0/2994452880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08419f450 con 0x7ff084101e40 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.082+0000 7ff080ff9640 1 -- 172.21.15.38:0/2994452880 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07c0042a0 con 0x7ff084101e40 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.082+0000 7ff089dd1640 1 -- 172.21.15.38:0/2994452880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08419f940 con 0x7ff084101e40 2024-03-26T06:35:35.087 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.082+0000 7ff089dd1640 1 -- 172.21.15.38:0/2994452880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08419fba0 con 0x7ff084101e40 2024-03-26T06:35:35.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.084+0000 7ff080ff9640 1 -- 172.21.15.38:0/2994452880 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff07c003970 con 0x7ff084101e40 2024-03-26T06:35:35.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.084+0000 7ff080ff9640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff06c0a2890 0x7ff06c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:35.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.084+0000 7ff0837fe640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff06c0a2890 0x7ff06c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:35.088 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.084+0000 7ff080ff9640 1 -- 172.21.15.38:0/2994452880 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff07c0f1530 con 0x7ff084101e40 2024-03-26T06:35:35.089 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.085+0000 7ff0837fe640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff06c0a2890 0x7ff06c0a4d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff0740058c0 tx=0x7ff074005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:35.090 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.088+0000 7ff080ff9640 1 -- 172.21.15.38:0/2994452880 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff07c0be7c0 con 0x7ff084101e40 2024-03-26T06:35:35.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.269+0000 7ff089dd1640 1 -- 172.21.15.38:0/2994452880 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff0840fd950 con 0x7ff06c0a2890 2024-03-26T06:35:35.279 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.277+0000 7ff080ff9640 1 -- 172.21.15.38:0/2994452880 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258709 (secure 0 0 0) 0x7ff0840fd950 con 0x7ff06c0a2890 2024-03-26T06:35:35.282 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:35.283 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-26T06:35:35.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 -- 172.21.15.38:0/2994452880 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff06c0a2890 msgr2=0x7ff06c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:35.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff06c0a2890 0x7ff06c0a4d50 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff0740058c0 tx=0x7ff074005d50 comp rx=0 tx=0).stop 2024-03-26T06:35:35.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 -- 172.21.15.38:0/2994452880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff084101e40 msgr2=0x7ff08419eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:35.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff084101e40 0x7ff08419eb50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff07c02c700 tx=0x7ff07c032f20 comp rx=0 tx=0).stop 2024-03-26T06:35:35.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 -- 172.21.15.38:0/2994452880 shutdown_connections 2024-03-26T06:35:35.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff06c0a2890 0x7ff06c0a4d50 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:35.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff084101e40 0x7ff08419eb50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:35.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff084101480 0x7ff084197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:35.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 --2- 172.21.15.38:0/2994452880 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff084100290 0x7ff0841975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:35.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 -- 172.21.15.38:0/2994452880 >> 172.21.15.38:0/2994452880 conn(0x7ff0840fb820 msgr2=0x7ff0840fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:35.285 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 -- 172.21.15.38:0/2994452880 shutdown_connections 2024-03-26T06:35:35.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:35.280+0000 7ff089dd1640 1 -- 172.21.15.38:0/2994452880 wait complete. 2024-03-26T06:35:35.293 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-03-26T06:35:34.234705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593580,"num_objects":40,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":261,"num_write_kb":2962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":244,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370216,"kb_used_data":12704,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124571608,"statfs":{"total":1151940427776,"available":1151561326592,"internally_reserved":0,"allocated":13008896,"data_stored":7080934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3212,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001677"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388036+0000","last_change":"2024-03-26T06:35:27.137165+0000","last_active":"2024-03-26T06:35:28.388036+0000","last_peered":"2024-03-26T06:35:28.388036+0000","last_clean":"2024-03-26T06:35:28.388036+0000","last_became_active":"2024-03-26T06:35:27.136869+0000","last_became_peered":"2024-03-26T06:35:27.136869+0000","last_unstale":"2024-03-26T06:35:28.388036+0000","last_undegraded":"2024-03-26T06:35:28.388036+0000","last_fullsized":"2024-03-26T06:35:28.388036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:43:36.671967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387117+0000","last_change":"2024-03-26T06:35:27.130619+0000","last_active":"2024-03-26T06:35:28.387117+0000","last_peered":"2024-03-26T06:35:28.387117+0000","last_clean":"2024-03-26T06:35:28.387117+0000","last_became_active":"2024-03-26T06:35:27.128846+0000","last_became_peered":"2024-03-26T06:35:27.128846+0000","last_unstale":"2024-03-26T06:35:28.387117+0000","last_undegraded":"2024-03-26T06:35:28.387117+0000","last_fullsized":"2024-03-26T06:35:28.387117+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:14:15.627268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387152+0000","last_change":"2024-03-26T06:35:27.138214+0000","last_active":"2024-03-26T06:35:28.387152+0000","last_peered":"2024-03-26T06:35:28.387152+0000","last_clean":"2024-03-26T06:35:28.387152+0000","last_became_active":"2024-03-26T06:35:27.137846+0000","last_became_peered":"2024-03-26T06:35:27.137846+0000","last_unstale":"2024-03-26T06:35:28.387152+0000","last_undegraded":"2024-03-26T06:35:28.387152+0000","last_fullsized":"2024-03-26T06:35:28.387152+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:48:46.441889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124881+0000","last_change":"2024-03-26T06:35:27.129582+0000","last_active":"2024-03-26T06:35:28.124881+0000","last_peered":"2024-03-26T06:35:28.124881+0000","last_clean":"2024-03-26T06:35:28.124881+0000","last_became_active":"2024-03-26T06:35:27.129245+0000","last_became_peered":"2024-03-26T06:35:27.129245+0000","last_unstale":"2024-03-26T06:35:28.124881+0000","last_undegraded":"2024-03-26T06:35:28.124881+0000","last_fullsized":"2024-03-26T06:35:28.124881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:09:14.548067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315409+0000","last_change":"2024-03-26T06:35:27.131993+0000","last_active":"2024-03-26T06:35:28.315409+0000","last_peered":"2024-03-26T06:35:28.315409+0000","last_clean":"2024-03-26T06:35:28.315409+0000","last_became_active":"2024-03-26T06:35:27.131650+0000","last_became_peered":"2024-03-26T06:35:27.131650+0000","last_unstale":"2024-03-26T06:35:28.315409+0000","last_undegraded":"2024-03-26T06:35:28.315409+0000","last_fullsized":"2024-03-26T06:35:28.315409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:38:41.407314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388036+0000","last_change":"2024-03-26T06:35:27.130325+0000","last_active":"2024-03-26T06:35:28.388036+0000","last_peered":"2024-03-26T06:35:28.388036+0000","last_clean":"2024-03-26T06:35:28.388036+0000","last_became_active":"2024-03-26T06:35:27.129658+0000","last_became_peered":"2024-03-26T06:35:27.129658+0000","last_unstale":"2024-03-26T06:35:28.388036+0000","last_undegraded":"2024-03-26T06:35:28.388036+0000","last_fullsized":"2024-03-26T06:35:28.388036+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:25:18.580839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387686+0000","last_change":"2024-03-26T06:35:27.130870+0000","last_active":"2024-03-26T06:35:28.387686+0000","last_peered":"2024-03-26T06:35:28.387686+0000","last_clean":"2024-03-26T06:35:28.387686+0000","last_became_active":"2024-03-26T06:35:27.130521+0000","last_became_peered":"2024-03-26T06:35:27.130521+0000","last_unstale":"2024-03-26T06:35:28.387686+0000","last_undegraded":"2024-03-26T06:35:28.387686+0000","last_fullsized":"2024-03-26T06:35:28.387686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:58:40.640708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125807+0000","last_change":"2024-03-26T06:35:27.136258+0000","last_active":"2024-03-26T06:35:28.125807+0000","last_peered":"2024-03-26T06:35:28.125807+0000","last_clean":"2024-03-26T06:35:28.125807+0000","last_became_active":"2024-03-26T06:35:27.135746+0000","last_became_peered":"2024-03-26T06:35:27.135746+0000","last_unstale":"2024-03-26T06:35:28.125807+0000","last_undegraded":"2024-03-26T06:35:28.125807+0000","last_fullsized":"2024-03-26T06:35:28.125807+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:42:24.270717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388694+0000","last_change":"2024-03-26T06:35:27.130533+0000","last_active":"2024-03-26T06:35:28.388694+0000","last_peered":"2024-03-26T06:35:28.388694+0000","last_clean":"2024-03-26T06:35:28.388694+0000","last_became_active":"2024-03-26T06:35:27.130251+0000","last_became_peered":"2024-03-26T06:35:27.130251+0000","last_unstale":"2024-03-26T06:35:28.388694+0000","last_undegraded":"2024-03-26T06:35:28.388694+0000","last_fullsized":"2024-03-26T06:35:28.388694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T18:13:40.209389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388817+0000","last_change":"2024-03-26T06:35:27.130494+0000","last_active":"2024-03-26T06:35:28.388817+0000","last_peered":"2024-03-26T06:35:28.388817+0000","last_clean":"2024-03-26T06:35:28.388817+0000","last_became_active":"2024-03-26T06:35:27.130048+0000","last_became_peered":"2024-03-26T06:35:27.130048+0000","last_unstale":"2024-03-26T06:35:28.388817+0000","last_undegraded":"2024-03-26T06:35:28.388817+0000","last_fullsized":"2024-03-26T06:35:28.388817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:34:19.649515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388336+0000","last_change":"2024-03-26T06:35:27.137285+0000","last_active":"2024-03-26T06:35:28.388336+0000","last_peered":"2024-03-26T06:35:28.388336+0000","last_clean":"2024-03-26T06:35:28.388336+0000","last_became_active":"2024-03-26T06:35:27.136930+0000","last_became_peered":"2024-03-26T06:35:27.136930+0000","last_unstale":"2024-03-26T06:35:28.388336+0000","last_undegraded":"2024-03-26T06:35:28.388336+0000","last_fullsized":"2024-03-26T06:35:28.388336+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:36:42.036591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124923+0000","last_change":"2024-03-26T06:35:27.133279+0000","last_active":"2024-03-26T06:35:28.124923+0000","last_peered":"2024-03-26T06:35:28.124923+0000","last_clean":"2024-03-26T06:35:28.124923+0000","last_became_active":"2024-03-26T06:35:27.132860+0000","last_became_peered":"2024-03-26T06:35:27.132860+0000","last_unstale":"2024-03-26T06:35:28.124923+0000","last_undegraded":"2024-03-26T06:35:28.124923+0000","last_fullsized":"2024-03-26T06:35:28.124923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:36:34.455411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387010+0000","last_change":"2024-03-26T06:35:27.124143+0000","last_active":"2024-03-26T06:35:28.387010+0000","last_peered":"2024-03-26T06:35:28.387010+0000","last_clean":"2024-03-26T06:35:28.387010+0000","last_became_active":"2024-03-26T06:35:27.123774+0000","last_became_peered":"2024-03-26T06:35:27.123774+0000","last_unstale":"2024-03-26T06:35:28.387010+0000","last_undegraded":"2024-03-26T06:35:28.387010+0000","last_fullsized":"2024-03-26T06:35:28.387010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:05:14.135531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387457+0000","last_change":"2024-03-26T06:35:27.127523+0000","last_active":"2024-03-26T06:35:28.387457+0000","last_peered":"2024-03-26T06:35:28.387457+0000","last_clean":"2024-03-26T06:35:28.387457+0000","last_became_active":"2024-03-26T06:35:27.127141+0000","last_became_peered":"2024-03-26T06:35:27.127141+0000","last_unstale":"2024-03-26T06:35:28.387457+0000","last_undegraded":"2024-03-26T06:35:28.387457+0000","last_fullsized":"2024-03-26T06:35:28.387457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:21:44.524721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.386959+0000","last_change":"2024-03-26T06:35:27.131744+0000","last_active":"2024-03-26T06:35:28.386959+0000","last_peered":"2024-03-26T06:35:28.386959+0000","last_clean":"2024-03-26T06:35:28.386959+0000","last_became_active":"2024-03-26T06:35:27.128735+0000","last_became_peered":"2024-03-26T06:35:27.128735+0000","last_unstale":"2024-03-26T06:35:28.386959+0000","last_undegraded":"2024-03-26T06:35:28.386959+0000","last_fullsized":"2024-03-26T06:35:28.386959+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:08:18.457258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.386999+0000","last_change":"2024-03-26T06:35:27.132021+0000","last_active":"2024-03-26T06:35:28.386999+0000","last_peered":"2024-03-26T06:35:28.386999+0000","last_clean":"2024-03-26T06:35:28.386999+0000","last_became_active":"2024-03-26T06:35:27.131722+0000","last_became_peered":"2024-03-26T06:35:27.131722+0000","last_unstale":"2024-03-26T06:35:28.386999+0000","last_undegraded":"2024-03-26T06:35:28.386999+0000","last_fullsized":"2024-03-26T06:35:28.386999+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:52:23.253250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388227+0000","last_change":"2024-03-26T06:35:27.137225+0000","last_active":"2024-03-26T06:35:28.388227+0000","last_peered":"2024-03-26T06:35:28.388227+0000","last_clean":"2024-03-26T06:35:28.388227+0000","last_became_active":"2024-03-26T06:35:27.136935+0000","last_became_peered":"2024-03-26T06:35:27.136935+0000","last_unstale":"2024-03-26T06:35:28.388227+0000","last_undegraded":"2024-03-26T06:35:28.388227+0000","last_fullsized":"2024-03-26T06:35:28.388227+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:14:02.540482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.313039+0000","last_change":"2024-03-26T06:35:27.136068+0000","last_active":"2024-03-26T06:35:28.313039+0000","last_peered":"2024-03-26T06:35:28.313039+0000","last_clean":"2024-03-26T06:35:28.313039+0000","last_became_active":"2024-03-26T06:35:27.135586+0000","last_became_peered":"2024-03-26T06:35:27.135586+0000","last_unstale":"2024-03-26T06:35:28.313039+0000","last_undegraded":"2024-03-26T06:35:28.313039+0000","last_fullsized":"2024-03-26T06:35:28.313039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:21:51.729675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387550+0000","last_change":"2024-03-26T06:35:27.131576+0000","last_active":"2024-03-26T06:35:28.387550+0000","last_peered":"2024-03-26T06:35:28.387550+0000","last_clean":"2024-03-26T06:35:28.387550+0000","last_became_active":"2024-03-26T06:35:27.131262+0000","last_became_peered":"2024-03-26T06:35:27.131262+0000","last_unstale":"2024-03-26T06:35:28.387550+0000","last_undegraded":"2024-03-26T06:35:28.387550+0000","last_fullsized":"2024-03-26T06:35:28.387550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:50:13.688512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:31.170968+0000","last_change":"2024-03-26T06:35:26.106217+0000","last_active":"2024-03-26T06:35:31.170968+0000","last_peered":"2024-03-26T06:35:31.170968+0000","last_clean":"2024-03-26T06:35:31.170968+0000","last_became_active":"2024-03-26T06:35:26.105866+0000","last_became_peered":"2024-03-26T06:35:26.105866+0000","last_unstale":"2024-03-26T06:35:31.170968+0000","last_undegraded":"2024-03-26T06:35:31.170968+0000","last_fullsized":"2024-03-26T06:35:31.170968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:52:04.770844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126319+0000","last_change":"2024-03-26T06:35:27.141510+0000","last_active":"2024-03-26T06:35:28.126319+0000","last_peered":"2024-03-26T06:35:28.126319+0000","last_clean":"2024-03-26T06:35:28.126319+0000","last_became_active":"2024-03-26T06:35:27.141182+0000","last_became_peered":"2024-03-26T06:35:27.141182+0000","last_unstale":"2024-03-26T06:35:28.126319+0000","last_undegraded":"2024-03-26T06:35:28.126319+0000","last_fullsized":"2024-03-26T06:35:28.126319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:40:55.993939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126585+0000","last_change":"2024-03-26T06:35:26.095502+0000","last_active":"2024-03-26T06:35:28.126585+0000","last_peered":"2024-03-26T06:35:28.126585+0000","last_clean":"2024-03-26T06:35:28.126585+0000","last_became_active":"2024-03-26T06:35:26.095168+0000","last_became_peered":"2024-03-26T06:35:26.095168+0000","last_unstale":"2024-03-26T06:35:28.126585+0000","last_undegraded":"2024-03-26T06:35:28.126585+0000","last_fullsized":"2024-03-26T06:35:28.126585+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:01:44.307274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387572+0000","last_change":"2024-03-26T06:35:27.137179+0000","last_active":"2024-03-26T06:35:28.387572+0000","last_peered":"2024-03-26T06:35:28.387572+0000","last_clean":"2024-03-26T06:35:28.387572+0000","last_became_active":"2024-03-26T06:35:27.136660+0000","last_became_peered":"2024-03-26T06:35:27.136660+0000","last_unstale":"2024-03-26T06:35:28.387572+0000","last_undegraded":"2024-03-26T06:35:28.387572+0000","last_fullsized":"2024-03-26T06:35:28.387572+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:16:16.042134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315749+0000","last_change":"2024-03-26T06:35:26.105395+0000","last_active":"2024-03-26T06:35:28.315749+0000","last_peered":"2024-03-26T06:35:28.315749+0000","last_clean":"2024-03-26T06:35:28.315749+0000","last_became_active":"2024-03-26T06:35:26.104879+0000","last_became_peered":"2024-03-26T06:35:26.104879+0000","last_unstale":"2024-03-26T06:35:28.315749+0000","last_undegraded":"2024-03-26T06:35:28.315749+0000","last_fullsized":"2024-03-26T06:35:28.315749+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:44:57.444844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387205+0000","last_change":"2024-03-26T06:35:27.133549+0000","last_active":"2024-03-26T06:35:28.387205+0000","last_peered":"2024-03-26T06:35:28.387205+0000","last_clean":"2024-03-26T06:35:28.387205+0000","last_became_active":"2024-03-26T06:35:27.132176+0000","last_became_peered":"2024-03-26T06:35:27.132176+0000","last_unstale":"2024-03-26T06:35:28.387205+0000","last_undegraded":"2024-03-26T06:35:28.387205+0000","last_fullsized":"2024-03-26T06:35:28.387205+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:15:59.281268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315597+0000","last_change":"2024-03-26T06:35:26.101438+0000","last_active":"2024-03-26T06:35:28.315597+0000","last_peered":"2024-03-26T06:35:28.315597+0000","last_clean":"2024-03-26T06:35:28.315597+0000","last_became_active":"2024-03-26T06:35:26.101139+0000","last_became_peered":"2024-03-26T06:35:26.101139+0000","last_unstale":"2024-03-26T06:35:28.315597+0000","last_undegraded":"2024-03-26T06:35:28.315597+0000","last_fullsized":"2024-03-26T06:35:28.315597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:10:36.750046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125198+0000","last_change":"2024-03-26T06:35:27.127896+0000","last_active":"2024-03-26T06:35:28.125198+0000","last_peered":"2024-03-26T06:35:28.125198+0000","last_clean":"2024-03-26T06:35:28.125198+0000","last_became_active":"2024-03-26T06:35:27.127558+0000","last_became_peered":"2024-03-26T06:35:27.127558+0000","last_unstale":"2024-03-26T06:35:28.125198+0000","last_undegraded":"2024-03-26T06:35:28.125198+0000","last_fullsized":"2024-03-26T06:35:28.125198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:04:58.033255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124716+0000","last_change":"2024-03-26T06:35:26.100148+0000","last_active":"2024-03-26T06:35:28.124716+0000","last_peered":"2024-03-26T06:35:28.124716+0000","last_clean":"2024-03-26T06:35:28.124716+0000","last_became_active":"2024-03-26T06:35:26.099827+0000","last_became_peered":"2024-03-26T06:35:26.099827+0000","last_unstale":"2024-03-26T06:35:28.124716+0000","last_undegraded":"2024-03-26T06:35:28.124716+0000","last_fullsized":"2024-03-26T06:35:28.124716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:33:21.845866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315457+0000","last_change":"2024-03-26T06:35:27.125318+0000","last_active":"2024-03-26T06:35:28.315457+0000","last_peered":"2024-03-26T06:35:28.315457+0000","last_clean":"2024-03-26T06:35:28.315457+0000","last_became_active":"2024-03-26T06:35:27.125013+0000","last_became_peered":"2024-03-26T06:35:27.125013+0000","last_unstale":"2024-03-26T06:35:28.315457+0000","last_undegraded":"2024-03-26T06:35:28.315457+0000","last_fullsized":"2024-03-26T06:35:28.315457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:50:25.123317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387754+0000","last_change":"2024-03-26T06:35:26.097150+0000","last_active":"2024-03-26T06:35:28.387754+0000","last_peered":"2024-03-26T06:35:28.387754+0000","last_clean":"2024-03-26T06:35:28.387754+0000","last_became_active":"2024-03-26T06:35:26.096844+0000","last_became_peered":"2024-03-26T06:35:26.096844+0000","last_unstale":"2024-03-26T06:35:28.387754+0000","last_undegraded":"2024-03-26T06:35:28.387754+0000","last_fullsized":"2024-03-26T06:35:28.387754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:57:47.399493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124891+0000","last_change":"2024-03-26T06:35:27.132261+0000","last_active":"2024-03-26T06:35:28.124891+0000","last_peered":"2024-03-26T06:35:28.124891+0000","last_clean":"2024-03-26T06:35:28.124891+0000","last_became_active":"2024-03-26T06:35:27.131893+0000","last_became_peered":"2024-03-26T06:35:27.131893+0000","last_unstale":"2024-03-26T06:35:28.124891+0000","last_undegraded":"2024-03-26T06:35:28.124891+0000","last_fullsized":"2024-03-26T06:35:28.124891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:42.521515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.386661+0000","last_change":"2024-03-26T06:35:26.092356+0000","last_active":"2024-03-26T06:35:28.386661+0000","last_peered":"2024-03-26T06:35:28.386661+0000","last_clean":"2024-03-26T06:35:28.386661+0000","last_became_active":"2024-03-26T06:35:26.091948+0000","last_became_peered":"2024-03-26T06:35:26.091948+0000","last_unstale":"2024-03-26T06:35:28.386661+0000","last_undegraded":"2024-03-26T06:35:28.386661+0000","last_fullsized":"2024-03-26T06:35:28.386661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:48:47.011884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125923+0000","last_change":"2024-03-26T06:35:27.129880+0000","last_active":"2024-03-26T06:35:28.125923+0000","last_peered":"2024-03-26T06:35:28.125923+0000","last_clean":"2024-03-26T06:35:28.125923+0000","last_became_active":"2024-03-26T06:35:27.129513+0000","last_became_peered":"2024-03-26T06:35:27.129513+0000","last_unstale":"2024-03-26T06:35:28.125923+0000","last_undegraded":"2024-03-26T06:35:28.125923+0000","last_fullsized":"2024-03-26T06:35:28.125923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:10:53.747657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124504+0000","last_change":"2024-03-26T06:35:26.095603+0000","last_active":"2024-03-26T06:35:28.124504+0000","last_peered":"2024-03-26T06:35:28.124504+0000","last_clean":"2024-03-26T06:35:28.124504+0000","last_became_active":"2024-03-26T06:35:26.095309+0000","last_became_peered":"2024-03-26T06:35:26.095309+0000","last_unstale":"2024-03-26T06:35:28.124504+0000","last_undegraded":"2024-03-26T06:35:28.124504+0000","last_fullsized":"2024-03-26T06:35:28.124504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:30:55.373598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.312889+0000","last_change":"2024-03-26T06:35:27.133271+0000","last_active":"2024-03-26T06:35:28.312889+0000","last_peered":"2024-03-26T06:35:28.312889+0000","last_clean":"2024-03-26T06:35:28.312889+0000","last_became_active":"2024-03-26T06:35:27.132553+0000","last_became_peered":"2024-03-26T06:35:27.132553+0000","last_unstale":"2024-03-26T06:35:28.312889+0000","last_undegraded":"2024-03-26T06:35:28.312889+0000","last_fullsized":"2024-03-26T06:35:28.312889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:13:59.021131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.158550+0000","last_change":"2024-03-26T06:35:26.094323+0000","last_active":"2024-03-26T06:35:30.158550+0000","last_peered":"2024-03-26T06:35:30.158550+0000","last_clean":"2024-03-26T06:35:30.158550+0000","last_became_active":"2024-03-26T06:35:26.093814+0000","last_became_peered":"2024-03-26T06:35:26.093814+0000","last_unstale":"2024-03-26T06:35:30.158550+0000","last_undegraded":"2024-03-26T06:35:30.158550+0000","last_fullsized":"2024-03-26T06:35:30.158550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:11.450427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388143+0000","last_change":"2024-03-26T06:35:27.127398+0000","last_active":"2024-03-26T06:35:28.388143+0000","last_peered":"2024-03-26T06:35:28.388143+0000","last_clean":"2024-03-26T06:35:28.388143+0000","last_became_active":"2024-03-26T06:35:27.127054+0000","last_became_peered":"2024-03-26T06:35:27.127054+0000","last_unstale":"2024-03-26T06:35:28.388143+0000","last_undegraded":"2024-03-26T06:35:28.388143+0000","last_fullsized":"2024-03-26T06:35:28.388143+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:51:04.150954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387996+0000","last_change":"2024-03-26T06:35:26.098531+0000","last_active":"2024-03-26T06:35:28.387996+0000","last_peered":"2024-03-26T06:35:28.387996+0000","last_clean":"2024-03-26T06:35:28.387996+0000","last_became_active":"2024-03-26T06:35:26.097942+0000","last_became_peered":"2024-03-26T06:35:26.097942+0000","last_unstale":"2024-03-26T06:35:28.387996+0000","last_undegraded":"2024-03-26T06:35:28.387996+0000","last_fullsized":"2024-03-26T06:35:28.387996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:44:59.884627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387508+0000","last_change":"2024-03-26T06:35:27.136145+0000","last_active":"2024-03-26T06:35:28.387508+0000","last_peered":"2024-03-26T06:35:28.387508+0000","last_clean":"2024-03-26T06:35:28.387508+0000","last_became_active":"2024-03-26T06:35:27.135560+0000","last_became_peered":"2024-03-26T06:35:27.135560+0000","last_unstale":"2024-03-26T06:35:28.387508+0000","last_undegraded":"2024-03-26T06:35:28.387508+0000","last_fullsized":"2024-03-26T06:35:28.387508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:19:01.991175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"85'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125211+0000","last_change":"2024-03-26T06:35:26.095119+0000","last_active":"2024-03-26T06:35:28.125211+0000","last_peered":"2024-03-26T06:35:28.125211+0000","last_clean":"2024-03-26T06:35:28.125211+0000","last_became_active":"2024-03-26T06:35:26.094789+0000","last_became_peered":"2024-03-26T06:35:26.094789+0000","last_unstale":"2024-03-26T06:35:28.125211+0000","last_undegraded":"2024-03-26T06:35:28.125211+0000","last_fullsized":"2024-03-26T06:35:28.125211+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:59:07.626207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387410+0000","last_change":"2024-03-26T06:35:27.133722+0000","last_active":"2024-03-26T06:35:28.387410+0000","last_peered":"2024-03-26T06:35:28.387410+0000","last_clean":"2024-03-26T06:35:28.387410+0000","last_became_active":"2024-03-26T06:35:27.132313+0000","last_became_peered":"2024-03-26T06:35:27.132313+0000","last_unstale":"2024-03-26T06:35:28.387410+0000","last_undegraded":"2024-03-26T06:35:28.387410+0000","last_fullsized":"2024-03-26T06:35:28.387410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:58:48.350708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.158968+0000","last_change":"2024-03-26T06:35:26.095229+0000","last_active":"2024-03-26T06:35:30.158968+0000","last_peered":"2024-03-26T06:35:30.158968+0000","last_clean":"2024-03-26T06:35:30.158968+0000","last_became_active":"2024-03-26T06:35:26.094929+0000","last_became_peered":"2024-03-26T06:35:26.094929+0000","last_unstale":"2024-03-26T06:35:30.158968+0000","last_undegraded":"2024-03-26T06:35:30.158968+0000","last_fullsized":"2024-03-26T06:35:30.158968+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:49:33.080958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315596+0000","last_change":"2024-03-26T06:35:27.127735+0000","last_active":"2024-03-26T06:35:28.315596+0000","last_peered":"2024-03-26T06:35:28.315596+0000","last_clean":"2024-03-26T06:35:28.315596+0000","last_became_active":"2024-03-26T06:35:27.127434+0000","last_became_peered":"2024-03-26T06:35:27.127434+0000","last_unstale":"2024-03-26T06:35:28.315596+0000","last_undegraded":"2024-03-26T06:35:28.315596+0000","last_fullsized":"2024-03-26T06:35:28.315596+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:30:09.745415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.158641+0000","last_change":"2024-03-26T06:35:26.095943+0000","last_active":"2024-03-26T06:35:30.158641+0000","last_peered":"2024-03-26T06:35:30.158641+0000","last_clean":"2024-03-26T06:35:30.158641+0000","last_became_active":"2024-03-26T06:35:26.095577+0000","last_became_peered":"2024-03-26T06:35:26.095577+0000","last_unstale":"2024-03-26T06:35:30.158641+0000","last_undegraded":"2024-03-26T06:35:30.158641+0000","last_fullsized":"2024-03-26T06:35:30.158641+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:42:34.459051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.386780+0000","last_change":"2024-03-26T06:35:27.133199+0000","last_active":"2024-03-26T06:35:28.386780+0000","last_peered":"2024-03-26T06:35:28.386780+0000","last_clean":"2024-03-26T06:35:28.386780+0000","last_became_active":"2024-03-26T06:35:27.132187+0000","last_became_peered":"2024-03-26T06:35:27.132187+0000","last_unstale":"2024-03-26T06:35:28.386780+0000","last_undegraded":"2024-03-26T06:35:28.386780+0000","last_fullsized":"2024-03-26T06:35:28.386780+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:16:22.832804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124833+0000","last_change":"2024-03-26T06:35:26.100902+0000","last_active":"2024-03-26T06:35:28.124833+0000","last_peered":"2024-03-26T06:35:28.124833+0000","last_clean":"2024-03-26T06:35:28.124833+0000","last_became_active":"2024-03-26T06:35:26.100509+0000","last_became_peered":"2024-03-26T06:35:26.100509+0000","last_unstale":"2024-03-26T06:35:28.124833+0000","last_undegraded":"2024-03-26T06:35:28.124833+0000","last_fullsized":"2024-03-26T06:35:28.124833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:52:53.459610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388014+0000","last_change":"2024-03-26T06:35:27.129126+0000","last_active":"2024-03-26T06:35:28.388014+0000","last_peered":"2024-03-26T06:35:28.388014+0000","last_clean":"2024-03-26T06:35:28.388014+0000","last_became_active":"2024-03-26T06:35:27.128741+0000","last_became_peered":"2024-03-26T06:35:27.128741+0000","last_unstale":"2024-03-26T06:35:28.388014+0000","last_undegraded":"2024-03-26T06:35:28.388014+0000","last_fullsized":"2024-03-26T06:35:28.388014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:09:15.613369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:31.168340+0000","last_change":"2024-03-26T06:35:26.096239+0000","last_active":"2024-03-26T06:35:31.168340+0000","last_peered":"2024-03-26T06:35:31.168340+0000","last_clean":"2024-03-26T06:35:31.168340+0000","last_became_active":"2024-03-26T06:35:26.095915+0000","last_became_peered":"2024-03-26T06:35:26.095915+0000","last_unstale":"2024-03-26T06:35:31.168340+0000","last_undegraded":"2024-03-26T06:35:31.168340+0000","last_fullsized":"2024-03-26T06:35:31.168340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:50:24.566322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388526+0000","last_change":"2024-03-26T06:35:27.125846+0000","last_active":"2024-03-26T06:35:28.388526+0000","last_peered":"2024-03-26T06:35:28.388526+0000","last_clean":"2024-03-26T06:35:28.388526+0000","last_became_active":"2024-03-26T06:35:27.125515+0000","last_became_peered":"2024-03-26T06:35:27.125515+0000","last_unstale":"2024-03-26T06:35:28.388526+0000","last_undegraded":"2024-03-26T06:35:28.388526+0000","last_fullsized":"2024-03-26T06:35:28.388526+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:02:17.358507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"85'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315156+0000","last_change":"2024-03-26T06:35:26.102857+0000","last_active":"2024-03-26T06:35:28.315156+0000","last_peered":"2024-03-26T06:35:28.315156+0000","last_clean":"2024-03-26T06:35:28.315156+0000","last_became_active":"2024-03-26T06:35:26.102552+0000","last_became_peered":"2024-03-26T06:35:26.102552+0000","last_unstale":"2024-03-26T06:35:28.315156+0000","last_undegraded":"2024-03-26T06:35:28.315156+0000","last_fullsized":"2024-03-26T06:35:28.315156+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:31:30.833012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125177+0000","last_change":"2024-03-26T06:35:27.133496+0000","last_active":"2024-03-26T06:35:28.125177+0000","last_peered":"2024-03-26T06:35:28.125177+0000","last_clean":"2024-03-26T06:35:28.125177+0000","last_became_active":"2024-03-26T06:35:27.133162+0000","last_became_peered":"2024-03-26T06:35:27.133162+0000","last_unstale":"2024-03-26T06:35:28.125177+0000","last_undegraded":"2024-03-26T06:35:28.125177+0000","last_fullsized":"2024-03-26T06:35:28.125177+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:00:48.927846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315675+0000","last_change":"2024-03-26T06:35:26.103876+0000","last_active":"2024-03-26T06:35:28.315675+0000","last_peered":"2024-03-26T06:35:28.315675+0000","last_clean":"2024-03-26T06:35:28.315675+0000","last_became_active":"2024-03-26T06:35:26.102981+0000","last_became_peered":"2024-03-26T06:35:26.102981+0000","last_unstale":"2024-03-26T06:35:28.315675+0000","last_undegraded":"2024-03-26T06:35:28.315675+0000","last_fullsized":"2024-03-26T06:35:28.315675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:24:04.154212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387576+0000","last_change":"2024-03-26T06:35:27.127529+0000","last_active":"2024-03-26T06:35:28.387576+0000","last_peered":"2024-03-26T06:35:28.387576+0000","last_clean":"2024-03-26T06:35:28.387576+0000","last_became_active":"2024-03-26T06:35:27.127208+0000","last_became_peered":"2024-03-26T06:35:27.127208+0000","last_unstale":"2024-03-26T06:35:28.387576+0000","last_undegraded":"2024-03-26T06:35:28.387576+0000","last_fullsized":"2024-03-26T06:35:28.387576+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:29:35.313042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.312949+0000","last_change":"2024-03-26T06:35:26.099309+0000","last_active":"2024-03-26T06:35:28.312949+0000","last_peered":"2024-03-26T06:35:28.312949+0000","last_clean":"2024-03-26T06:35:28.312949+0000","last_became_active":"2024-03-26T06:35:26.098948+0000","last_became_peered":"2024-03-26T06:35:26.098948+0000","last_unstale":"2024-03-26T06:35:28.312949+0000","last_undegraded":"2024-03-26T06:35:28.312949+0000","last_fullsized":"2024-03-26T06:35:28.312949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:49:35.210018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388763+0000","last_change":"2024-03-26T06:35:27.132441+0000","last_active":"2024-03-26T06:35:28.388763+0000","last_peered":"2024-03-26T06:35:28.388763+0000","last_clean":"2024-03-26T06:35:28.388763+0000","last_became_active":"2024-03-26T06:35:27.131846+0000","last_became_peered":"2024-03-26T06:35:27.131846+0000","last_unstale":"2024-03-26T06:35:28.388763+0000","last_undegraded":"2024-03-26T06:35:28.388763+0000","last_fullsized":"2024-03-26T06:35:28.388763+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:14.779299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387970+0000","last_change":"2024-03-26T06:35:26.107214+0000","last_active":"2024-03-26T06:35:28.387970+0000","last_peered":"2024-03-26T06:35:28.387970+0000","last_clean":"2024-03-26T06:35:28.387970+0000","last_became_active":"2024-03-26T06:35:26.106935+0000","last_became_peered":"2024-03-26T06:35:26.106935+0000","last_unstale":"2024-03-26T06:35:28.387970+0000","last_undegraded":"2024-03-26T06:35:28.387970+0000","last_fullsized":"2024-03-26T06:35:28.387970+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:49:33.101354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388280+0000","last_change":"2024-03-26T06:35:27.131374+0000","last_active":"2024-03-26T06:35:28.388280+0000","last_peered":"2024-03-26T06:35:28.388280+0000","last_clean":"2024-03-26T06:35:28.388280+0000","last_became_active":"2024-03-26T06:35:27.131030+0000","last_became_peered":"2024-03-26T06:35:27.131030+0000","last_unstale":"2024-03-26T06:35:28.388280+0000","last_undegraded":"2024-03-26T06:35:28.388280+0000","last_fullsized":"2024-03-26T06:35:28.388280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:33:40.809278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124706+0000","last_change":"2024-03-26T06:35:27.127240+0000","last_active":"2024-03-26T06:35:28.124706+0000","last_peered":"2024-03-26T06:35:28.124706+0000","last_clean":"2024-03-26T06:35:28.124706+0000","last_became_active":"2024-03-26T06:35:27.126923+0000","last_became_peered":"2024-03-26T06:35:27.126923+0000","last_unstale":"2024-03-26T06:35:28.124706+0000","last_undegraded":"2024-03-26T06:35:28.124706+0000","last_fullsized":"2024-03-26T06:35:28.124706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:28:20.422536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.386604+0000","last_change":"2024-03-26T06:35:26.096718+0000","last_active":"2024-03-26T06:35:28.386604+0000","last_peered":"2024-03-26T06:35:28.386604+0000","last_clean":"2024-03-26T06:35:28.386604+0000","last_became_active":"2024-03-26T06:35:26.096407+0000","last_became_peered":"2024-03-26T06:35:26.096407+0000","last_unstale":"2024-03-26T06:35:28.386604+0000","last_undegraded":"2024-03-26T06:35:28.386604+0000","last_fullsized":"2024-03-26T06:35:28.386604+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:12:00.799567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126374+0000","last_change":"2024-03-26T06:35:27.138485+0000","last_active":"2024-03-26T06:35:28.126374+0000","last_peered":"2024-03-26T06:35:28.126374+0000","last_clean":"2024-03-26T06:35:28.126374+0000","last_became_active":"2024-03-26T06:35:27.138114+0000","last_became_peered":"2024-03-26T06:35:27.138114+0000","last_unstale":"2024-03-26T06:35:28.126374+0000","last_undegraded":"2024-03-26T06:35:28.126374+0000","last_fullsized":"2024-03-26T06:35:28.126374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:28:25.244683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.157548+0000","last_change":"2024-03-26T06:35:26.096111+0000","last_active":"2024-03-26T06:35:30.157548+0000","last_peered":"2024-03-26T06:35:30.157548+0000","last_clean":"2024-03-26T06:35:30.157548+0000","last_became_active":"2024-03-26T06:35:26.095809+0000","last_became_peered":"2024-03-26T06:35:26.095809+0000","last_unstale":"2024-03-26T06:35:30.157548+0000","last_undegraded":"2024-03-26T06:35:30.157548+0000","last_fullsized":"2024-03-26T06:35:30.157548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:04:17.436961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":348,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388229+0000","last_change":"2024-03-26T06:34:13.886034+0000","last_active":"2024-03-26T06:35:28.388229+0000","last_peered":"2024-03-26T06:35:28.388229+0000","last_clean":"2024-03-26T06:35:28.388229+0000","last_became_active":"2024-03-26T06:34:13.885472+0000","last_became_peered":"2024-03-26T06:34:13.885472+0000","last_unstale":"2024-03-26T06:35:28.388229+0000","last_undegraded":"2024-03-26T06:35:28.388229+0000","last_fullsized":"2024-03-26T06:35:28.388229+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:30:52.854577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:30:52.854577+0000","last_clean_scrub_stamp":"2024-03-26T06:30:52.854577+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:43:35.315474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387189+0000","last_change":"2024-03-26T06:35:27.131778+0000","last_active":"2024-03-26T06:35:28.387189+0000","last_peered":"2024-03-26T06:35:28.387189+0000","last_clean":"2024-03-26T06:35:28.387189+0000","last_became_active":"2024-03-26T06:35:27.131433+0000","last_became_peered":"2024-03-26T06:35:27.131433+0000","last_unstale":"2024-03-26T06:35:28.387189+0000","last_undegraded":"2024-03-26T06:35:28.387189+0000","last_fullsized":"2024-03-26T06:35:28.387189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:17:58.614371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388726+0000","last_change":"2024-03-26T06:35:26.106052+0000","last_active":"2024-03-26T06:35:28.388726+0000","last_peered":"2024-03-26T06:35:28.388726+0000","last_clean":"2024-03-26T06:35:28.388726+0000","last_became_active":"2024-03-26T06:35:26.105728+0000","last_became_peered":"2024-03-26T06:35:26.105728+0000","last_unstale":"2024-03-26T06:35:28.388726+0000","last_undegraded":"2024-03-26T06:35:28.388726+0000","last_fullsized":"2024-03-26T06:35:28.388726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:19:12.226026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315564+0000","last_change":"2024-03-26T06:35:26.099487+0000","last_active":"2024-03-26T06:35:28.315564+0000","last_peered":"2024-03-26T06:35:28.315564+0000","last_clean":"2024-03-26T06:35:28.315564+0000","last_became_active":"2024-03-26T06:35:26.099180+0000","last_became_peered":"2024-03-26T06:35:26.099180+0000","last_unstale":"2024-03-26T06:35:28.315564+0000","last_undegraded":"2024-03-26T06:35:28.315564+0000","last_fullsized":"2024-03-26T06:35:28.315564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:15:20.832451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387142+0000","last_change":"2024-03-26T06:35:27.125799+0000","last_active":"2024-03-26T06:35:28.387142+0000","last_peered":"2024-03-26T06:35:28.387142+0000","last_clean":"2024-03-26T06:35:28.387142+0000","last_became_active":"2024-03-26T06:35:27.125323+0000","last_became_peered":"2024-03-26T06:35:27.125323+0000","last_unstale":"2024-03-26T06:35:28.387142+0000","last_undegraded":"2024-03-26T06:35:28.387142+0000","last_fullsized":"2024-03-26T06:35:28.387142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:07:31.162007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:31.170858+0000","last_change":"2024-03-26T06:35:26.103348+0000","last_active":"2024-03-26T06:35:31.170858+0000","last_peered":"2024-03-26T06:35:31.170858+0000","last_clean":"2024-03-26T06:35:31.170858+0000","last_became_active":"2024-03-26T06:35:26.103025+0000","last_became_peered":"2024-03-26T06:35:26.103025+0000","last_unstale":"2024-03-26T06:35:31.170858+0000","last_undegraded":"2024-03-26T06:35:31.170858+0000","last_fullsized":"2024-03-26T06:35:31.170858+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:51:51.581237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126431+0000","last_change":"2024-03-26T06:35:27.128853+0000","last_active":"2024-03-26T06:35:28.126431+0000","last_peered":"2024-03-26T06:35:28.126431+0000","last_clean":"2024-03-26T06:35:28.126431+0000","last_became_active":"2024-03-26T06:35:27.128472+0000","last_became_peered":"2024-03-26T06:35:27.128472+0000","last_unstale":"2024-03-26T06:35:28.126431+0000","last_undegraded":"2024-03-26T06:35:28.126431+0000","last_fullsized":"2024-03-26T06:35:28.126431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:01:01.574623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.157678+0000","last_change":"2024-03-26T06:35:26.100523+0000","last_active":"2024-03-26T06:35:30.157678+0000","last_peered":"2024-03-26T06:35:30.157678+0000","last_clean":"2024-03-26T06:35:30.157678+0000","last_became_active":"2024-03-26T06:35:26.099313+0000","last_became_peered":"2024-03-26T06:35:26.099313+0000","last_unstale":"2024-03-26T06:35:30.157678+0000","last_undegraded":"2024-03-26T06:35:30.157678+0000","last_fullsized":"2024-03-26T06:35:30.157678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:43:44.182464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387380+0000","last_change":"2024-03-26T06:35:27.131438+0000","last_active":"2024-03-26T06:35:28.387380+0000","last_peered":"2024-03-26T06:35:28.387380+0000","last_clean":"2024-03-26T06:35:28.387380+0000","last_became_active":"2024-03-26T06:35:27.131112+0000","last_became_peered":"2024-03-26T06:35:27.131112+0000","last_unstale":"2024-03-26T06:35:28.387380+0000","last_undegraded":"2024-03-26T06:35:28.387380+0000","last_fullsized":"2024-03-26T06:35:28.387380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:07.621191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.157557+0000","last_change":"2024-03-26T06:35:26.105039+0000","last_active":"2024-03-26T06:35:30.157557+0000","last_peered":"2024-03-26T06:35:30.157557+0000","last_clean":"2024-03-26T06:35:30.157557+0000","last_became_active":"2024-03-26T06:35:26.104776+0000","last_became_peered":"2024-03-26T06:35:26.104776+0000","last_unstale":"2024-03-26T06:35:30.157557+0000","last_undegraded":"2024-03-26T06:35:30.157557+0000","last_fullsized":"2024-03-26T06:35:30.157557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:20:17.748144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315283+0000","last_change":"2024-03-26T06:35:27.126320+0000","last_active":"2024-03-26T06:35:28.315283+0000","last_peered":"2024-03-26T06:35:28.315283+0000","last_clean":"2024-03-26T06:35:28.315283+0000","last_became_active":"2024-03-26T06:35:27.125996+0000","last_became_peered":"2024-03-26T06:35:27.125996+0000","last_unstale":"2024-03-26T06:35:28.315283+0000","last_undegraded":"2024-03-26T06:35:28.315283+0000","last_fullsized":"2024-03-26T06:35:28.315283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:08:21.386787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:30.158403+0000","last_change":"2024-03-26T06:35:26.103303+0000","last_active":"2024-03-26T06:35:30.158403+0000","last_peered":"2024-03-26T06:35:30.158403+0000","last_clean":"2024-03-26T06:35:30.158403+0000","last_became_active":"2024-03-26T06:35:26.102311+0000","last_became_peered":"2024-03-26T06:35:26.102311+0000","last_unstale":"2024-03-26T06:35:30.158403+0000","last_undegraded":"2024-03-26T06:35:30.158403+0000","last_fullsized":"2024-03-26T06:35:30.158403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:44:13.845386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387347+0000","last_change":"2024-03-26T06:35:27.128845+0000","last_active":"2024-03-26T06:35:28.387347+0000","last_peered":"2024-03-26T06:35:28.387347+0000","last_clean":"2024-03-26T06:35:28.387347+0000","last_became_active":"2024-03-26T06:35:27.128532+0000","last_became_peered":"2024-03-26T06:35:27.128532+0000","last_unstale":"2024-03-26T06:35:28.387347+0000","last_undegraded":"2024-03-26T06:35:28.387347+0000","last_fullsized":"2024-03-26T06:35:28.387347+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:35:08.087689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387442+0000","last_change":"2024-03-26T06:35:26.107589+0000","last_active":"2024-03-26T06:35:28.387442+0000","last_peered":"2024-03-26T06:35:28.387442+0000","last_clean":"2024-03-26T06:35:28.387442+0000","last_became_active":"2024-03-26T06:35:26.107280+0000","last_became_peered":"2024-03-26T06:35:26.107280+0000","last_unstale":"2024-03-26T06:35:28.387442+0000","last_undegraded":"2024-03-26T06:35:28.387442+0000","last_fullsized":"2024-03-26T06:35:28.387442+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:57:27.621484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126155+0000","last_change":"2024-03-26T06:35:27.131238+0000","last_active":"2024-03-26T06:35:28.126155+0000","last_peered":"2024-03-26T06:35:28.126155+0000","last_clean":"2024-03-26T06:35:28.126155+0000","last_became_active":"2024-03-26T06:35:27.130902+0000","last_became_peered":"2024-03-26T06:35:27.130902+0000","last_unstale":"2024-03-26T06:35:28.126155+0000","last_undegraded":"2024-03-26T06:35:28.126155+0000","last_fullsized":"2024-03-26T06:35:28.126155+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:02:47.180199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.315271+0000","last_change":"2024-03-26T06:35:26.105886+0000","last_active":"2024-03-26T06:35:28.315271+0000","last_peered":"2024-03-26T06:35:28.315271+0000","last_clean":"2024-03-26T06:35:28.315271+0000","last_became_active":"2024-03-26T06:35:26.105429+0000","last_became_peered":"2024-03-26T06:35:26.105429+0000","last_unstale":"2024-03-26T06:35:28.315271+0000","last_undegraded":"2024-03-26T06:35:28.315271+0000","last_fullsized":"2024-03-26T06:35:28.315271+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:51:57.897588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388160+0000","last_change":"2024-03-26T06:35:27.133273+0000","last_active":"2024-03-26T06:35:28.388160+0000","last_peered":"2024-03-26T06:35:28.388160+0000","last_clean":"2024-03-26T06:35:28.388160+0000","last_became_active":"2024-03-26T06:35:27.132955+0000","last_became_peered":"2024-03-26T06:35:27.132955+0000","last_unstale":"2024-03-26T06:35:28.388160+0000","last_undegraded":"2024-03-26T06:35:28.388160+0000","last_fullsized":"2024-03-26T06:35:28.388160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:26:01.542473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387802+0000","last_change":"2024-03-26T06:35:26.103065+0000","last_active":"2024-03-26T06:35:28.387802+0000","last_peered":"2024-03-26T06:35:28.387802+0000","last_clean":"2024-03-26T06:35:28.387802+0000","last_became_active":"2024-03-26T06:35:26.102761+0000","last_became_peered":"2024-03-26T06:35:26.102761+0000","last_unstale":"2024-03-26T06:35:28.387802+0000","last_undegraded":"2024-03-26T06:35:28.387802+0000","last_fullsized":"2024-03-26T06:35:28.387802+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:09:27.650223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388509+0000","last_change":"2024-03-26T06:35:27.137820+0000","last_active":"2024-03-26T06:35:28.388509+0000","last_peered":"2024-03-26T06:35:28.388509+0000","last_clean":"2024-03-26T06:35:28.388509+0000","last_became_active":"2024-03-26T06:35:27.137375+0000","last_became_peered":"2024-03-26T06:35:27.137375+0000","last_unstale":"2024-03-26T06:35:28.388509+0000","last_undegraded":"2024-03-26T06:35:28.388509+0000","last_fullsized":"2024-03-26T06:35:28.388509+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:43:13.908662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124984+0000","last_change":"2024-03-26T06:35:26.094295+0000","last_active":"2024-03-26T06:35:28.124984+0000","last_peered":"2024-03-26T06:35:28.124984+0000","last_clean":"2024-03-26T06:35:28.124984+0000","last_became_active":"2024-03-26T06:35:26.093950+0000","last_became_peered":"2024-03-26T06:35:26.093950+0000","last_unstale":"2024-03-26T06:35:28.124984+0000","last_undegraded":"2024-03-26T06:35:28.124984+0000","last_fullsized":"2024-03-26T06:35:28.124984+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:45:23.700237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388195+0000","last_change":"2024-03-26T06:35:27.131253+0000","last_active":"2024-03-26T06:35:28.388195+0000","last_peered":"2024-03-26T06:35:28.388195+0000","last_clean":"2024-03-26T06:35:28.388195+0000","last_became_active":"2024-03-26T06:35:27.130929+0000","last_became_peered":"2024-03-26T06:35:27.130929+0000","last_unstale":"2024-03-26T06:35:28.388195+0000","last_undegraded":"2024-03-26T06:35:28.388195+0000","last_fullsized":"2024-03-26T06:35:28.388195+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:45:55.744139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387106+0000","last_change":"2024-03-26T06:35:26.104272+0000","last_active":"2024-03-26T06:35:28.387106+0000","last_peered":"2024-03-26T06:35:28.387106+0000","last_clean":"2024-03-26T06:35:28.387106+0000","last_became_active":"2024-03-26T06:35:26.103957+0000","last_became_peered":"2024-03-26T06:35:26.103957+0000","last_unstale":"2024-03-26T06:35:28.387106+0000","last_undegraded":"2024-03-26T06:35:28.387106+0000","last_fullsized":"2024-03-26T06:35:28.387106+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:09:29.037044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.388594+0000","last_change":"2024-03-26T06:35:27.128221+0000","last_active":"2024-03-26T06:35:28.388594+0000","last_peered":"2024-03-26T06:35:28.388594+0000","last_clean":"2024-03-26T06:35:28.388594+0000","last_became_active":"2024-03-26T06:35:27.126609+0000","last_became_peered":"2024-03-26T06:35:27.126609+0000","last_unstale":"2024-03-26T06:35:28.388594+0000","last_undegraded":"2024-03-26T06:35:28.388594+0000","last_fullsized":"2024-03-26T06:35:28.388594+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:59:56.314853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387544+0000","last_change":"2024-03-26T06:35:27.122330+0000","last_active":"2024-03-26T06:35:28.387544+0000","last_peered":"2024-03-26T06:35:28.387544+0000","last_clean":"2024-03-26T06:35:28.387544+0000","last_became_active":"2024-03-26T06:35:27.121989+0000","last_became_peered":"2024-03-26T06:35:27.121989+0000","last_unstale":"2024-03-26T06:35:28.387544+0000","last_undegraded":"2024-03-26T06:35:28.387544+0000","last_fullsized":"2024-03-26T06:35:28.387544+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:51:29.611555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387710+0000","last_change":"2024-03-26T06:35:27.127975+0000","last_active":"2024-03-26T06:35:28.387710+0000","last_peered":"2024-03-26T06:35:28.387710+0000","last_clean":"2024-03-26T06:35:28.387710+0000","last_became_active":"2024-03-26T06:35:27.127635+0000","last_became_peered":"2024-03-26T06:35:27.127635+0000","last_unstale":"2024-03-26T06:35:28.387710+0000","last_undegraded":"2024-03-26T06:35:28.387710+0000","last_fullsized":"2024-03-26T06:35:28.387710+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:48.587117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387732+0000","last_change":"2024-03-26T06:35:27.133538+0000","last_active":"2024-03-26T06:35:28.387732+0000","last_peered":"2024-03-26T06:35:28.387732+0000","last_clean":"2024-03-26T06:35:28.387732+0000","last_became_active":"2024-03-26T06:35:27.133104+0000","last_became_peered":"2024-03-26T06:35:27.133104+0000","last_unstale":"2024-03-26T06:35:28.387732+0000","last_undegraded":"2024-03-26T06:35:28.387732+0000","last_fullsized":"2024-03-26T06:35:28.387732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:20:10.417567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387515+0000","last_change":"2024-03-26T06:35:27.129503+0000","last_active":"2024-03-26T06:35:28.387515+0000","last_peered":"2024-03-26T06:35:28.387515+0000","last_clean":"2024-03-26T06:35:28.387515+0000","last_became_active":"2024-03-26T06:35:27.128930+0000","last_became_peered":"2024-03-26T06:35:27.128930+0000","last_unstale":"2024-03-26T06:35:28.387515+0000","last_undegraded":"2024-03-26T06:35:28.387515+0000","last_fullsized":"2024-03-26T06:35:28.387515+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:23:37.476593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387061+0000","last_change":"2024-03-26T06:35:27.137385+0000","last_active":"2024-03-26T06:35:28.387061+0000","last_peered":"2024-03-26T06:35:28.387061+0000","last_clean":"2024-03-26T06:35:28.387061+0000","last_became_active":"2024-03-26T06:35:27.136969+0000","last_became_peered":"2024-03-26T06:35:27.136969+0000","last_unstale":"2024-03-26T06:35:28.387061+0000","last_undegraded":"2024-03-26T06:35:28.387061+0000","last_fullsized":"2024-03-26T06:35:28.387061+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:40:16.722381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387451+0000","last_change":"2024-03-26T06:35:27.125287+0000","last_active":"2024-03-26T06:35:28.387451+0000","last_peered":"2024-03-26T06:35:28.387451+0000","last_clean":"2024-03-26T06:35:28.387451+0000","last_became_active":"2024-03-26T06:35:27.124274+0000","last_became_peered":"2024-03-26T06:35:27.124274+0000","last_unstale":"2024-03-26T06:35:28.387451+0000","last_undegraded":"2024-03-26T06:35:28.387451+0000","last_fullsized":"2024-03-26T06:35:28.387451+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:33:43.461949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387965+0000","last_change":"2024-03-26T06:35:27.124805+0000","last_active":"2024-03-26T06:35:28.387965+0000","last_peered":"2024-03-26T06:35:28.387965+0000","last_clean":"2024-03-26T06:35:28.387965+0000","last_became_active":"2024-03-26T06:35:27.123947+0000","last_became_peered":"2024-03-26T06:35:27.123947+0000","last_unstale":"2024-03-26T06:35:28.387965+0000","last_undegraded":"2024-03-26T06:35:28.387965+0000","last_fullsized":"2024-03-26T06:35:28.387965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:47.273083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.126289+0000","last_change":"2024-03-26T06:35:27.141167+0000","last_active":"2024-03-26T06:35:28.126289+0000","last_peered":"2024-03-26T06:35:28.126289+0000","last_clean":"2024-03-26T06:35:28.126289+0000","last_became_active":"2024-03-26T06:35:27.140781+0000","last_became_peered":"2024-03-26T06:35:27.140781+0000","last_unstale":"2024-03-26T06:35:28.126289+0000","last_undegraded":"2024-03-26T06:35:28.126289+0000","last_fullsized":"2024-03-26T06:35:28.126289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:42:36.705671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.124686+0000","last_change":"2024-03-26T06:35:27.132336+0000","last_active":"2024-03-26T06:35:28.124686+0000","last_peered":"2024-03-26T06:35:28.124686+0000","last_clean":"2024-03-26T06:35:28.124686+0000","last_became_active":"2024-03-26T06:35:27.131944+0000","last_became_peered":"2024-03-26T06:35:27.131944+0000","last_unstale":"2024-03-26T06:35:28.124686+0000","last_undegraded":"2024-03-26T06:35:28.124686+0000","last_fullsized":"2024-03-26T06:35:28.124686+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:51:57.916873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.125876+0000","last_change":"2024-03-26T06:35:27.141297+0000","last_active":"2024-03-26T06:35:28.125876+0000","last_peered":"2024-03-26T06:35:28.125876+0000","last_clean":"2024-03-26T06:35:28.125876+0000","last_became_active":"2024-03-26T06:35:27.140932+0000","last_became_peered":"2024-03-26T06:35:27.140932+0000","last_unstale":"2024-03-26T06:35:28.125876+0000","last_undegraded":"2024-03-26T06:35:28.125876+0000","last_fullsized":"2024-03-26T06:35:28.125876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:17.942885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387211+0000","last_change":"2024-03-26T06:35:27.135381+0000","last_active":"2024-03-26T06:35:28.387211+0000","last_peered":"2024-03-26T06:35:28.387211+0000","last_clean":"2024-03-26T06:35:28.387211+0000","last_became_active":"2024-03-26T06:35:27.135053+0000","last_became_peered":"2024-03-26T06:35:27.135053+0000","last_unstale":"2024-03-26T06:35:28.387211+0000","last_undegraded":"2024-03-26T06:35:28.387211+0000","last_fullsized":"2024-03-26T06:35:28.387211+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:40:21.960182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387033+0000","last_change":"2024-03-26T06:35:27.130780+0000","last_active":"2024-03-26T06:35:28.387033+0000","last_peered":"2024-03-26T06:35:28.387033+0000","last_clean":"2024-03-26T06:35:28.387033+0000","last_became_active":"2024-03-26T06:35:27.130171+0000","last_became_peered":"2024-03-26T06:35:27.130171+0000","last_unstale":"2024-03-26T06:35:28.387033+0000","last_undegraded":"2024-03-26T06:35:28.387033+0000","last_fullsized":"2024-03-26T06:35:28.387033+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:53:18.053617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-26T06:35:28.387779+0000","last_change":"2024-03-26T06:35:27.129677+0000","last_active":"2024-03-26T06:35:28.387779+0000","last_peered":"2024-03-26T06:35:28.387779+0000","last_clean":"2024-03-26T06:35:28.387779+0000","last_became_active":"2024-03-26T06:35:27.129275+0000","last_became_peered":"2024-03-26T06:35:27.129275+0000","last_unstale":"2024-03-26T06:35:28.387779+0000","last_undegraded":"2024-03-26T06:35:28.387779+0000","last_fullsized":"2024-03-26T06:35:28.387779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:18.851198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3212,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":80,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":11622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":442154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28532,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716620,"statfs":{"total":95995035648,"available":95965818880,"internally_reserved":0,"allocated":1552384,"data_stored":1034508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776151,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":442356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037788,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":929792,"data_stored":442176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299425,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32672,"kb_used_data":1484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712480,"statfs":{"total":95995035648,"available":95961579520,"internally_reserved":0,"allocated":1519616,"data_stored":1032522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462951,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":441572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574912,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":443184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738437,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":443050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40424,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704728,"statfs":{"total":95995035648,"available":95953641472,"internally_reserved":0,"allocated":946176,"data_stored":443840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149750,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":441856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888113,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":441594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724587,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28520,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716632,"statfs":{"total":95995035648,"available":95965831168,"internally_reserved":0,"allocated":1527808,"data_stored":1032122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T06:35:35.297 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-26T06:35:35.297 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T06:35:35.297 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T06:35:35.573 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:35.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:35 smithi038 ceph-mon[32265]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:35.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:35 smithi038 ceph-mon[32265]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:35.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:35 smithi038 ceph-mon[32265]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:35.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:35 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3051817298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:35.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:35 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1437537962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:35.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:35 smithi121 ceph-mon[36349]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:35.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:35 smithi121 ceph-mon[36349]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:35.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:35 smithi121 ceph-mon[36349]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:35.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:35 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3051817298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:35.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:35 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1437537962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:35.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:35 smithi170 ceph-mon[36912]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:35.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:35 smithi170 ceph-mon[36912]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:35.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:35 smithi170 ceph-mon[36912]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:35:35.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:35 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3051817298' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:35:35.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:35 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1437537962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:36 smithi038 ceph-mon[32265]: pgmap v290: 97 pgs: 97 active+clean; 580 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-26T06:35:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:36 smithi038 ceph-mon[32265]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:35:36.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:36 smithi121 ceph-mon[36349]: pgmap v290: 97 pgs: 97 active+clean; 580 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-26T06:35:36.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:36 smithi121 ceph-mon[36349]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:35:36.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:36 smithi170 ceph-mon[36912]: pgmap v290: 97 pgs: 97 active+clean; 580 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-03-26T06:35:36.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:36 smithi170 ceph-mon[36912]: from='client.14778 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:35:37.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.186+0000 7f021d2dc700 1 -- 172.21.15.38:0/471829130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218101e30 msgr2=0x7f0218102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:37.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.186+0000 7f021d2dc700 1 --2- 172.21.15.38:0/471829130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218101e30 0x7f0218102250 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0208009990 tx=0x7f0208009ca0 comp rx=0 tx=0).stop 2024-03-26T06:35:37.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.186+0000 7f021d2dc700 1 -- 172.21.15.38:0/471829130 shutdown_connections 2024-03-26T06:35:37.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.186+0000 7f021d2dc700 1 --2- 172.21.15.38:0/471829130 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0218103950 0x7f021810a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:37.190 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.186+0000 7f021d2dc700 1 --2- 172.21.15.38:0/471829130 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218101e30 0x7f0218102250 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.187+0000 7f021d2dc700 1 --2- 172.21.15.38:0/471829130 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0218102f90 0x7f0218103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.187+0000 7f021d2dc700 1 -- 172.21.15.38:0/471829130 >> 172.21.15.38:0/471829130 conn(0x7f02180fd3f0 msgr2=0x7f02180ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.187+0000 7f021d2dc700 1 -- 172.21.15.38:0/471829130 shutdown_connections 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.187+0000 7f021d2dc700 1 -- 172.21.15.38:0/471829130 wait complete. 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.187+0000 7f021d2dc700 1 Processor -- start 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.187+0000 7f021d2dc700 1 -- start start 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.187+0000 7f021d2dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0218101e30 0x7f0218197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.187+0000 7f021d2dc700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0218102f90 0x7f02181984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f021d2dc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218103950 0x7f021819f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f021d2dc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021810da20 con 0x7f0218103950 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f021d2dc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021810d8a0 con 0x7f0218101e30 2024-03-26T06:35:37.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f021d2dc700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021810dba0 con 0x7f0218102f90 2024-03-26T06:35:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f02177fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218103950 0x7f021819f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f0216ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0218101e30 0x7f0218197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f02167fc700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0218102f90 0x7f02181984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f02177fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218103950 0x7f021819f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50114/0 (socket says 172.21.15.38:50114) 2024-03-26T06:35:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f0216ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0218101e30 0x7f0218197fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:45944/0 (socket says 172.21.15.38:45944) 2024-03-26T06:35:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f02177fe700 1 -- 172.21.15.38:0/2756169806 learned_addr learned my addr 172.21.15.38:0/2756169806 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f02177fe700 1 -- 172.21.15.38:0/2756169806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0218102f90 msgr2=0x7f02181984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f02177fe700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0218102f90 0x7f02181984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f02177fe700 1 -- 172.21.15.38:0/2756169806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0218101e30 msgr2=0x7f0218197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f02177fe700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0218101e30 0x7f0218197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:37.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f02177fe700 1 -- 172.21.15.38:0/2756169806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021819fc10 con 0x7f0218103950 2024-03-26T06:35:37.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f02177fe700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218103950 0x7f021819f570 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f021000ef60 tx=0x7f021000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:37.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f01fffff700 1 -- 172.21.15.38:0/2756169806 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02100092e0 con 0x7f0218103950 2024-03-26T06:35:37.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f01fffff700 1 -- 172.21.15.38:0/2756169806 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0210004bb0 con 0x7f0218103950 2024-03-26T06:35:37.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f021d2dc700 1 -- 172.21.15.38:0/2756169806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f021819fea0 con 0x7f0218103950 2024-03-26T06:35:37.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f01fffff700 1 -- 172.21.15.38:0/2756169806 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0210004280 con 0x7f0218103950 2024-03-26T06:35:37.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.188+0000 7f021d2dc700 1 -- 172.21.15.38:0/2756169806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02181a0300 con 0x7f0218103950 2024-03-26T06:35:37.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.189+0000 7f021d2dc700 1 -- 172.21.15.38:0/2756169806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02180502f0 con 0x7f0218103950 2024-03-26T06:35:37.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.190+0000 7f01fffff700 1 -- 172.21.15.38:0/2756169806 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f021001b030 con 0x7f0218103950 2024-03-26T06:35:37.194 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.191+0000 7f01fffff700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f02000a29d0 0x7f02000a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:37.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.191+0000 7f0216ffd700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f02000a29d0 0x7f02000a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:37.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.191+0000 7f01fffff700 1 -- 172.21.15.38:0/2756169806 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f02100ce450 con 0x7f0218103950 2024-03-26T06:35:37.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.191+0000 7f0216ffd700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f02000a29d0 0x7f02000a4e90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f020800b2e0 tx=0x7f020800b820 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:37.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.197+0000 7f01fffff700 1 -- 172.21.15.38:0/2756169806 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0210022020 con 0x7f0218103950 2024-03-26T06:35:37.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.489+0000 7f021d2dc700 1 -- 172.21.15.38:0/2756169806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f02180629f0 con 0x7f0218103950 2024-03-26T06:35:37.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:37 smithi038 ceph-mon[32265]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:35:37.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.496+0000 7f01fffff700 1 -- 172.21.15.38:0/2756169806 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f0210010040 con 0x7f0218103950 2024-03-26T06:35:37.498 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2024-03-26T06:35:37.498 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQC5bAJm4MxJHRAAsoKEJRuQMGditbmAH5jWRg== 2024-03-26T06:35:37.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 -- 172.21.15.38:0/2756169806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f02000a29d0 msgr2=0x7f02000a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:37.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f02000a29d0 0x7f02000a4e90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f020800b2e0 tx=0x7f020800b820 comp rx=0 tx=0).stop 2024-03-26T06:35:37.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 -- 172.21.15.38:0/2756169806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218103950 msgr2=0x7f021819f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:37.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218103950 0x7f021819f570 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f021000ef60 tx=0x7f021000ef90 comp rx=0 tx=0).stop 2024-03-26T06:35:37.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 -- 172.21.15.38:0/2756169806 shutdown_connections 2024-03-26T06:35:37.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0218103950 0x7f021819f570 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:37.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f02000a29d0 0x7f02000a4e90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:37.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0218101e30 0x7f0218197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:37.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 --2- 172.21.15.38:0/2756169806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0218102f90 0x7f02181984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:37.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 -- 172.21.15.38:0/2756169806 >> 172.21.15.38:0/2756169806 conn(0x7f02180fd3f0 msgr2=0x7f02181062c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:37.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 -- 172.21.15.38:0/2756169806 shutdown_connections 2024-03-26T06:35:37.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:37.499+0000 7f021d2dc700 1 -- 172.21.15.38:0/2756169806 wait complete. 2024-03-26T06:35:37.569 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:37 smithi170 ceph-mon[36912]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:35:37.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:37 smithi121 ceph-mon[36349]: from='client.14784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:35:38.127 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:35:38.127 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T06:35:38.127 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T06:35:38.172 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T06:35:38.173 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T06:35:38.173 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T06:35:38.456 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:38.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:38 smithi038 ceph-mon[32265]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.8 KiB/s wr, 19 op/s 2024-03-26T06:35:38.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:38 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2756169806' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T06:35:38.578 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:38 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2756169806' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T06:35:38.583 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:38 smithi170 ceph-mon[36912]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.8 KiB/s wr, 19 op/s 2024-03-26T06:35:38.583 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:38 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2756169806' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T06:35:38.583 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:38 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2756169806' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T06:35:38.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:38 smithi121 ceph-mon[36349]: pgmap v291: 97 pgs: 97 active+clean; 618 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.8 KiB/s wr, 19 op/s 2024-03-26T06:35:38.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:38 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2756169806' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T06:35:38.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:38 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2756169806' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T06:35:39.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:39 smithi038 ceph-mon[32265]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-03-26T06:35:39.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:39 smithi121 ceph-mon[36349]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-03-26T06:35:39.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:39 smithi170 ceph-mon[36912]: pgmap v292: 97 pgs: 97 active+clean; 618 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 8.0 KiB/s wr, 18 op/s 2024-03-26T06:35:40.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.015+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/10904993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f58101910 msgr2=0x7f7f58105960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.015+0000 7f7f5ccfb700 1 --2- 172.21.15.38:0/10904993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f58101910 0x7f7f58105960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0099c0 tx=0x7f7f4c009cd0 comp rx=0 tx=0).stop 2024-03-26T06:35:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.015+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/10904993 shutdown_connections 2024-03-26T06:35:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.015+0000 7f7f5ccfb700 1 --2- 172.21.15.38:0/10904993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f58106040 0x7f7f5810dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.015+0000 7f7f5ccfb700 1 --2- 172.21.15.38:0/10904993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f58100f60 0x7f7f58101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.015+0000 7f7f5ccfb700 1 --2- 172.21.15.38:0/10904993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f58101910 0x7f7f58105960 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.015+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/10904993 >> 172.21.15.38:0/10904993 conn(0x7f7f58077de0 msgr2=0x7f7f580781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:40.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.015+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/10904993 shutdown_connections 2024-03-26T06:35:40.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.015+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/10904993 wait complete. 2024-03-26T06:35:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5ccfb700 1 Processor -- start 2024-03-26T06:35:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5ccfb700 1 -- start start 2024-03-26T06:35:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5ccfb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f58100f60 0x7f7f581985e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5ccfb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f58101910 0x7f7f58198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5ccfb700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f58106040 0x7f7f5819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5ccfb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f58111080 con 0x7f7f58100f60 2024-03-26T06:35:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5ccfb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f58110f00 con 0x7f7f58101910 2024-03-26T06:35:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5ccfb700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f58111200 con 0x7f7f58106040 2024-03-26T06:35:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f58100f60 0x7f7f581985e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f58100f60 0x7f7f581985e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50140/0 (socket says 172.21.15.38:50140) 2024-03-26T06:35:40.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5659c700 1 -- 172.21.15.38:0/1432235597 learned_addr learned my addr 172.21.15.38:0/1432235597 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:40.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f56d9d700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f58106040 0x7f7f5819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:40.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f55d9b700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f58101910 0x7f7f58198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:40.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5659c700 1 -- 172.21.15.38:0/1432235597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f58106040 msgr2=0x7f7f5819ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:40.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5659c700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f58106040 0x7f7f5819ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:40.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5659c700 1 -- 172.21.15.38:0/1432235597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f58101910 msgr2=0x7f7f58198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:40.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5659c700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f58101910 0x7f7f58198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:40.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.016+0000 7f7f5659c700 1 -- 172.21.15.38:0/1432235597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5819d5f0 con 0x7f7f58100f60 2024-03-26T06:35:40.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.017+0000 7f7f5659c700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f58100f60 0x7f7f581985e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800b530 tx=0x7f7f4800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:40.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.017+0000 7f7f477fe700 1 -- 172.21.15.38:0/1432235597 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48015020 con 0x7f7f58100f60 2024-03-26T06:35:40.022 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.017+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/1432235597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5819d8e0 con 0x7f7f58100f60 2024-03-26T06:35:40.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.017+0000 7f7f477fe700 1 -- 172.21.15.38:0/1432235597 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f48011d10 con 0x7f7f58100f60 2024-03-26T06:35:40.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.017+0000 7f7f477fe700 1 -- 172.21.15.38:0/1432235597 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4800f2e0 con 0x7f7f58100f60 2024-03-26T06:35:40.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.017+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/1432235597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f58102930 con 0x7f7f58100f60 2024-03-26T06:35:40.023 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.018+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/1432235597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f58068aa0 con 0x7f7f58100f60 2024-03-26T06:35:40.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.019+0000 7f7f477fe700 1 -- 172.21.15.38:0/1432235597 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7f48018020 con 0x7f7f58100f60 2024-03-26T06:35:40.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.020+0000 7f7f477fe700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7f400a2920 0x7f7f400a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:40.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.020+0000 7f7f55d9b700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7f400a2920 0x7f7f400a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:40.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.020+0000 7f7f477fe700 1 -- 172.21.15.38:0/1432235597 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7f480ce110 con 0x7f7f58100f60 2024-03-26T06:35:40.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.020+0000 7f7f55d9b700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7f400a2920 0x7f7f400a4de0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009990 tx=0x7f7f4c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:40.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.024+0000 7f7f477fe700 1 -- 172.21.15.38:0/1432235597 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7f4801f020 con 0x7f7f58100f60 2024-03-26T06:35:40.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.288+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/1432235597 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f7f58199940 con 0x7f7f58100f60 2024-03-26T06:35:40.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.293+0000 7f7f477fe700 1 -- 172.21.15.38:0/1432235597 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262661 (secure 0 0 0) 0x7f7f4809b210 con 0x7f7f58100f60 2024-03-26T06:35:40.299 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:40.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/1432235597 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7f400a2920 msgr2=0x7f7f400a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:40.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7f400a2920 0x7f7f400a4de0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009990 tx=0x7f7f4c005fb0 comp rx=0 tx=0).stop 2024-03-26T06:35:40.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/1432235597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f58100f60 msgr2=0x7f7f581985e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:40.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f58100f60 0x7f7f581985e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7f4800b530 tx=0x7f7f4800b840 comp rx=0 tx=0).stop 2024-03-26T06:35:40.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/1432235597 shutdown_connections 2024-03-26T06:35:40.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7f58106040 0x7f7f5819ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:40.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7f58100f60 0x7f7f581985e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:40.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7f400a2920 0x7f7f400a4de0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:40.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 --2- 172.21.15.38:0/1432235597 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f58101910 0x7f7f58198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:40.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/1432235597 >> 172.21.15.38:0/1432235597 conn(0x7f7f58077de0 msgr2=0x7f7f5806a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:40.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/1432235597 shutdown_connections 2024-03-26T06:35:40.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:40.296+0000 7f7f5ccfb700 1 -- 172.21.15.38:0/1432235597 wait complete. 2024-03-26T06:35:40.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:40 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1432235597' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T06:35:40.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:40 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1432235597' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T06:35:40.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:40 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1432235597' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T06:35:40.977 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":2716021457},{"type":"v1","addr":"172.21.15.38:6801","nonce":2716021457}]},"active_addr":"172.21.15.38:6801/2716021457","active_change":"2024-03-26T06:28:26.137894+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.38:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":1675065388}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":3570258347}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":4027757314}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":3320689790}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2060337180}]}]} 2024-03-26T06:35:40.983 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T06:35:40.983 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T06:35:40.983 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T06:35:41.234 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:41.629 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:41 smithi038 ceph-mon[32265]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-03-26T06:35:41.644 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:41 smithi121 ceph-mon[36349]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-03-26T06:35:41.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:41 smithi170 ceph-mon[36912]: pgmap v293: 97 pgs: 97 active+clean; 618 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 6.7 KiB/s wr, 16 op/s 2024-03-26T06:35:42.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.958+0000 7fa1790e1700 1 -- 172.21.15.38:0/2315274488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa174105fc0 msgr2=0x7fa17410db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:42.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.958+0000 7fa1790e1700 1 --2- 172.21.15.38:0/2315274488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa174105fc0 0x7fa17410db90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa16c00b0d0 tx=0x7fa16c00b3e0 comp rx=0 tx=0).stop 2024-03-26T06:35:42.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.958+0000 7fa1790e1700 1 -- 172.21.15.38:0/2315274488 shutdown_connections 2024-03-26T06:35:42.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.958+0000 7fa1790e1700 1 --2- 172.21.15.38:0/2315274488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa174105fc0 0x7fa17410db90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:42.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.958+0000 7fa1790e1700 1 --2- 172.21.15.38:0/2315274488 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa174100f70 0x7fa174101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:42.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.958+0000 7fa1790e1700 1 --2- 172.21.15.38:0/2315274488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa174101920 0x7fa1741058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:42.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.958+0000 7fa1790e1700 1 -- 172.21.15.38:0/2315274488 >> 172.21.15.38:0/2315274488 conn(0x7fa174076d30 msgr2=0x7fa174077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:42.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.958+0000 7fa1790e1700 1 -- 172.21.15.38:0/2315274488 shutdown_connections 2024-03-26T06:35:42.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.958+0000 7fa1790e1700 1 -- 172.21.15.38:0/2315274488 wait complete. 2024-03-26T06:35:42.961 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa1790e1700 1 Processor -- start 2024-03-26T06:35:42.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa1790e1700 1 -- start start 2024-03-26T06:35:42.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa1790e1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa174100f70 0x7fa174102980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:42.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa1790e1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa174101920 0x7fa174102ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:42.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa1790e1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa174105fc0 0x7fa174103400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:42.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa1790e1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa174111020 con 0x7fa174105fc0 2024-03-26T06:35:42.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa1790e1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa174110ea0 con 0x7fa174101920 2024-03-26T06:35:42.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa1790e1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1741111a0 con 0x7fa174100f70 2024-03-26T06:35:42.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa17359e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa174105fc0 0x7fa174103400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:42.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa17359e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa174105fc0 0x7fa174103400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50166/0 (socket says 172.21.15.38:50166) 2024-03-26T06:35:42.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa17359e700 1 -- 172.21.15.38:0/2070109704 learned_addr learned my addr 172.21.15.38:0/2070109704 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:42.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa17259c700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa174101920 0x7fa174102ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:42.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.959+0000 7fa172d9d700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa174100f70 0x7fa174102980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa17359e700 1 -- 172.21.15.38:0/2070109704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa174100f70 msgr2=0x7fa174102980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa17359e700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa174100f70 0x7fa174102980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa17359e700 1 -- 172.21.15.38:0/2070109704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa174101920 msgr2=0x7fa174102ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa17359e700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa174101920 0x7fa174102ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa17359e700 1 -- 172.21.15.38:0/2070109704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1741a6ac0 con 0x7fa174105fc0 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa172d9d700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa174100f70 0x7fa174102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa17359e700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa174105fc0 0x7fa174103400 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa16c006070 tx=0x7fa16c004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa15bfff700 1 -- 172.21.15.38:0/2070109704 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c021070 con 0x7fa174105fc0 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa15bfff700 1 -- 172.21.15.38:0/2070109704 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa16c007d00 con 0x7fa174105fc0 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa1790e1700 1 -- 172.21.15.38:0/2070109704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1741a6d50 con 0x7fa174105fc0 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa15bfff700 1 -- 172.21.15.38:0/2070109704 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16c005470 con 0x7fa174105fc0 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.960+0000 7fa1790e1700 1 -- 172.21.15.38:0/2070109704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1741a7200 con 0x7fa174105fc0 2024-03-26T06:35:42.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.961+0000 7fa1790e1700 1 -- 172.21.15.38:0/2070109704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa174068aa0 con 0x7fa174105fc0 2024-03-26T06:35:42.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.962+0000 7fa15bfff700 1 -- 172.21.15.38:0/2070109704 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa16c024070 con 0x7fa174105fc0 2024-03-26T06:35:42.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.963+0000 7fa15bfff700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa15c0a2920 0x7fa15c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:42.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.963+0000 7fa172d9d700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa15c0a2920 0x7fa15c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:42.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.963+0000 7fa15bfff700 1 -- 172.21.15.38:0/2070109704 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa16c00f020 con 0x7fa174105fc0 2024-03-26T06:35:42.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.963+0000 7fa172d9d700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa15c0a2920 0x7fa15c0a4de0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa1680099d0 tx=0x7fa168008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:42.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:42.967+0000 7fa15bfff700 1 -- 172.21.15.38:0/2070109704 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa16c02b020 con 0x7fa174105fc0 2024-03-26T06:35:43.172 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.170+0000 7fa1790e1700 1 -- 172.21.15.38:0/2070109704 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa17402d9f0 con 0x7fa174105fc0 2024-03-26T06:35:43.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.171+0000 7fa15bfff700 1 -- 172.21.15.38:0/2070109704 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22209 (secure 0 0 0) 0x7fa16c09d270 con 0x7fa174105fc0 2024-03-26T06:35:43.174 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:43.174 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":86,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:35:28.120088+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 -- 172.21.15.38:0/2070109704 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa15c0a2920 msgr2=0x7fa15c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa15c0a2920 0x7fa15c0a4de0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa1680099d0 tx=0x7fa168008040 comp rx=0 tx=0).stop 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 -- 172.21.15.38:0/2070109704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa174105fc0 msgr2=0x7fa174103400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa174105fc0 0x7fa174103400 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa16c006070 tx=0x7fa16c004e40 comp rx=0 tx=0).stop 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 -- 172.21.15.38:0/2070109704 shutdown_connections 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa174105fc0 0x7fa174103400 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa174100f70 0x7fa174102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa15c0a2920 0x7fa15c0a4de0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 --2- 172.21.15.38:0/2070109704 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa174101920 0x7fa174102ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 -- 172.21.15.38:0/2070109704 >> 172.21.15.38:0/2070109704 conn(0x7fa174076d30 msgr2=0x7fa17406a600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 -- 172.21.15.38:0/2070109704 shutdown_connections 2024-03-26T06:35:43.176 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:43.173+0000 7fa1790e1700 1 -- 172.21.15.38:0/2070109704 wait complete. 2024-03-26T06:35:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:43 smithi121 ceph-mon[36349]: pgmap v294: 97 pgs: 97 active+clean; 623 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 7.1 KiB/s wr, 16 op/s 2024-03-26T06:35:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:43 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2070109704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:43.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:43 smithi170 ceph-mon[36912]: pgmap v294: 97 pgs: 97 active+clean; 623 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 7.1 KiB/s wr, 16 op/s 2024-03-26T06:35:43.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:43 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2070109704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:43.800 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:43 smithi038 ceph-mon[32265]: pgmap v294: 97 pgs: 97 active+clean; 623 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 7.1 KiB/s wr, 16 op/s 2024-03-26T06:35:43.800 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:43 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2070109704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:43.830 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T06:35:43.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T06:35:44.074 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:45.564 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:45 smithi038 ceph-mon[32265]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-03-26T06:35:45.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:45 smithi121 ceph-mon[36349]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-03-26T06:35:45.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:45 smithi170 ceph-mon[36912]: pgmap v295: 97 pgs: 97 active+clean; 623 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.3 KiB/s wr, 12 op/s 2024-03-26T06:35:45.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.795+0000 7f1bb8d29700 1 -- 172.21.15.38:0/3693519488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bb406af00 msgr2=0x7f1bb410ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:45.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.795+0000 7f1bb8d29700 1 --2- 172.21.15.38:0/3693519488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bb406af00 0x7f1bb410ec60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009990 tx=0x7f1bac009ca0 comp rx=0 tx=0).stop 2024-03-26T06:35:45.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.795+0000 7f1bb8d29700 1 -- 172.21.15.38:0/3693519488 shutdown_connections 2024-03-26T06:35:45.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.795+0000 7f1bb8d29700 1 --2- 172.21.15.38:0/3693519488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bb406af00 0x7f1bb410ec60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:45.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.795+0000 7f1bb8d29700 1 --2- 172.21.15.38:0/3693519488 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb4107e20 0x7f1bb4108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:45.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.795+0000 7f1bb8d29700 1 --2- 172.21.15.38:0/3693519488 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb406a560 0x7f1bb406a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:45.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.795+0000 7f1bb8d29700 1 -- 172.21.15.38:0/3693519488 >> 172.21.15.38:0/3693519488 conn(0x7f1bb4077c80 msgr2=0x7f1bb4078090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:45.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.795+0000 7f1bb8d29700 1 -- 172.21.15.38:0/3693519488 shutdown_connections 2024-03-26T06:35:45.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.795+0000 7f1bb8d29700 1 -- 172.21.15.38:0/3693519488 wait complete. 2024-03-26T06:35:45.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.796+0000 7f1bb8d29700 1 Processor -- start 2024-03-26T06:35:45.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.796+0000 7f1bb8d29700 1 -- start start 2024-03-26T06:35:45.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.796+0000 7f1bb8d29700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb406a560 0x7f1bb4100da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.796+0000 7f1bb8d29700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bb406af00 0x7f1bb41012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.796+0000 7f1bb8d29700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb4107e20 0x7f1bb4101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.796+0000 7f1bb8d29700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb41120a0 con 0x7f1bb406af00 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.796+0000 7f1bb8d29700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4111f20 con 0x7f1bb406a560 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.796+0000 7f1bb8d29700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb4112220 con 0x7f1bb4107e20 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb1d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bb406af00 0x7f1bb41012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb1d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bb406af00 0x7f1bb41012e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:44440/0 (socket says 172.21.15.38:44440) 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb1d9b700 1 -- 172.21.15.38:0/1495959219 learned_addr learned my addr 172.21.15.38:0/1495959219 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb259c700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb406a560 0x7f1bb4100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb2d9d700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb4107e20 0x7f1bb4101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb1d9b700 1 -- 172.21.15.38:0/1495959219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb4107e20 msgr2=0x7f1bb4101820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb1d9b700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb4107e20 0x7f1bb4101820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:45.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb1d9b700 1 -- 172.21.15.38:0/1495959219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb406a560 msgr2=0x7f1bb4100da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:45.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb1d9b700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb406a560 0x7f1bb4100da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:45.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb1d9b700 1 -- 172.21.15.38:0/1495959219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb41a6a20 con 0x7f1bb406af00 2024-03-26T06:35:45.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb1d9b700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bb406af00 0x7f1bb41012e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00d950 tx=0x7f1b9c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:45.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bab7fe700 1 -- 172.21.15.38:0/1495959219 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c016070 con 0x7f1bb406af00 2024-03-26T06:35:45.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bab7fe700 1 -- 172.21.15.38:0/1495959219 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b9c009e20 con 0x7f1bb406af00 2024-03-26T06:35:45.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb8d29700 1 -- 172.21.15.38:0/1495959219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bb41a6d10 con 0x7f1bb406af00 2024-03-26T06:35:45.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bab7fe700 1 -- 172.21.15.38:0/1495959219 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9c00f620 con 0x7f1bb406af00 2024-03-26T06:35:45.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.797+0000 7f1bb8d29700 1 -- 172.21.15.38:0/1495959219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bb41a71f0 con 0x7f1bb406af00 2024-03-26T06:35:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.799+0000 7f1bb8d29700 1 -- 172.21.15.38:0/1495959219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb4068aa0 con 0x7f1bb406af00 2024-03-26T06:35:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.799+0000 7f1bab7fe700 1 -- 172.21.15.38:0/1495959219 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f1b9c019070 con 0x7f1bb406af00 2024-03-26T06:35:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.800+0000 7f1bab7fe700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1ba00a2920 0x7f1ba00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.800+0000 7f1bb259c700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1ba00a2920 0x7f1ba00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.800+0000 7f1bab7fe700 1 -- 172.21.15.38:0/1495959219 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1b9c0ce5e0 con 0x7f1bb406af00 2024-03-26T06:35:45.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.800+0000 7f1bb259c700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1ba00a2920 0x7f1ba00a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4007fd0 tx=0x7f1ba400c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:45.808 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:45.804+0000 7f1bab7fe700 1 -- 172.21.15.38:0/1495959219 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1b9c09b6d0 con 0x7f1bb406af00 2024-03-26T06:35:46.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.010+0000 7f1bb8d29700 1 -- 172.21.15.38:0/1495959219 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bb402d9f0 con 0x7f1bb406af00 2024-03-26T06:35:46.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.012+0000 7f1bab7fe700 1 -- 172.21.15.38:0/1495959219 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22209 (secure 0 0 0) 0x7f1b9c09afd0 con 0x7f1bb406af00 2024-03-26T06:35:46.015 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:35:46.016 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":86,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:35:28.120088+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:35:46.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 -- 172.21.15.38:0/1495959219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1ba00a2920 msgr2=0x7f1ba00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:46.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1ba00a2920 0x7f1ba00a4de0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4007fd0 tx=0x7f1ba400c040 comp rx=0 tx=0).stop 2024-03-26T06:35:46.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 -- 172.21.15.38:0/1495959219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bb406af00 msgr2=0x7f1bb41012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:46.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bb406af00 0x7f1bb41012e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00d950 tx=0x7f1b9c00dc60 comp rx=0 tx=0).stop 2024-03-26T06:35:46.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 -- 172.21.15.38:0/1495959219 shutdown_connections 2024-03-26T06:35:46.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1bb4107e20 0x7f1bb4101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:46.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1ba00a2920 0x7f1ba00a4de0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:46.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1bb406a560 0x7f1bb4100da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:46.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 --2- 172.21.15.38:0/1495959219 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1bb406af00 0x7f1bb41012e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:46.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 -- 172.21.15.38:0/1495959219 >> 172.21.15.38:0/1495959219 conn(0x7f1bb4077c80 msgr2=0x7f1bb40ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:46.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 -- 172.21.15.38:0/1495959219 shutdown_connections 2024-03-26T06:35:46.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:46.014+0000 7f1bb8d29700 1 -- 172.21.15.38:0/1495959219 wait complete. 2024-03-26T06:35:46.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T06:35:46.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T06:35:46.576 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T06:35:46.577 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T06:35:46.577 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T06:35:46.577 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T06:35:46.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T06:35:46.578 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T06:35:46.579 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-26T06:35:46.579 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-26T06:35:46.579 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-26T06:35:46.580 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-26T06:35:46.621 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:46 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1495959219' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:46.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:46 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1495959219' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:46.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:46 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1495959219' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:35:47.294 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.366 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.367 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.367 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.371 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.372 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.372 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.376 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.376 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.383 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:35:47.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:47 smithi121 ceph-mon[36349]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.8 KiB/s wr, 14 op/s 2024-03-26T06:35:47.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:47 smithi170 ceph-mon[36912]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.8 KiB/s wr, 14 op/s 2024-03-26T06:35:47.676 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:47 smithi038 ceph-mon[32265]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.8 KiB/s wr, 14 op/s 2024-03-26T06:35:49.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:49 smithi121 ceph-mon[36349]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-26T06:35:49.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:49 smithi170 ceph-mon[36912]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-26T06:35:49.685 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:49 smithi038 ceph-mon[32265]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-26T06:35:51.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:51 smithi121 ceph-mon[36349]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-26T06:35:51.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:51 smithi038 ceph-mon[32265]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-26T06:35:51.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:51 smithi170 ceph-mon[36912]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-03-26T06:35:53.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:53 smithi170 ceph-mon[36912]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-03-26T06:35:53.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:53 smithi038 ceph-mon[32265]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-03-26T06:35:53.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:53 smithi121 ceph-mon[36349]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2024-03-26T06:35:54.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.878+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1281656572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfa41020f0 msgr2=0x7fcfa4105980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:54.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.878+0000 7fcfa93fc700 1 --2- 172.21.15.38:0/1281656572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfa41020f0 0x7fcfa4105980 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcf980099c0 tx=0x7fcf98009cd0 comp rx=0 tx=0).stop 2024-03-26T06:35:54.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.878+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1281656572 shutdown_connections 2024-03-26T06:35:54.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.878+0000 7fcfa93fc700 1 --2- 172.21.15.38:0/1281656572 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa4106060 0x7fcfa410dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:54.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.878+0000 7fcfa93fc700 1 --2- 172.21.15.38:0/1281656572 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa4101740 0x7fcfa4101b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:54.882 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.878+0000 7fcfa93fc700 1 --2- 172.21.15.38:0/1281656572 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfa41020f0 0x7fcfa4105980 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.878+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1281656572 >> 172.21.15.38:0/1281656572 conn(0x7fcfa40772e0 msgr2=0x7fcfa40fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.878+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1281656572 shutdown_connections 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.878+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1281656572 wait complete. 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.878+0000 7fcfa93fc700 1 Processor -- start 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.878+0000 7fcfa93fc700 1 -- start start 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa93fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfa4101740 0x7fcfa4113420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa93fc700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa41020f0 0x7fcfa41117e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa93fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa4106060 0x7fcfa4111f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa93fc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa4111020 con 0x7fcfa4101740 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa93fc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa4110ea0 con 0x7fcfa4106060 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa93fc700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa41111a0 con 0x7fcfa41020f0 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfa4101740 0x7fcfa4113420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:54.883 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfa4101740 0x7fcfa4113420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47788/0 (socket says 172.21.15.38:47788) 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa2ffd700 1 -- 172.21.15.38:0/1112892940 learned_addr learned my addr 172.21.15.38:0/1112892940 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa27fc700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa41020f0 0x7fcfa41117e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa37fe700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa4106060 0x7fcfa4111f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa2ffd700 1 -- 172.21.15.38:0/1112892940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa41020f0 msgr2=0x7fcfa41117e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa2ffd700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa41020f0 0x7fcfa41117e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa2ffd700 1 -- 172.21.15.38:0/1112892940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa4106060 msgr2=0x7fcfa4111f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa2ffd700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcfa4106060 0x7fcfa4111f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa2ffd700 1 -- 172.21.15.38:0/1112892940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa41a6b80 con 0x7fcfa4101740 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa27fc700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcfa41020f0 0x7fcfa41117e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.879+0000 7fcfa2ffd700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfa4101740 0x7fcfa4113420 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400c920 tx=0x7fcf9400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.880+0000 7fcf8bfff700 1 -- 172.21.15.38:0/1112892940 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf94007b00 con 0x7fcfa4101740 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.880+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1112892940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa41a6db0 con 0x7fcfa4101740 2024-03-26T06:35:54.884 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.880+0000 7fcf8bfff700 1 -- 172.21.15.38:0/1112892940 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf94004d10 con 0x7fcfa4101740 2024-03-26T06:35:54.885 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.880+0000 7fcf8bfff700 1 -- 172.21.15.38:0/1112892940 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf940056e0 con 0x7fcfa4101740 2024-03-26T06:35:54.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.880+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1112892940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa41a71b0 con 0x7fcfa4101740 2024-03-26T06:35:54.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.880+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1112892940 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcfa41052b0 con 0x7fcfa4101740 2024-03-26T06:35:54.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.882+0000 7fcf8bfff700 1 -- 172.21.15.38:0/1112892940 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fcf94007ca0 con 0x7fcfa4101740 2024-03-26T06:35:54.886 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.882+0000 7fcf8bfff700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcf8c0a2920 0x7fcf8c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:54.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.882+0000 7fcfa27fc700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcf8c0a2920 0x7fcf8c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:54.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.882+0000 7fcf8bfff700 1 -- 172.21.15.38:0/1112892940 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcf940ceae0 con 0x7fcfa4101740 2024-03-26T06:35:54.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.883+0000 7fcf8bfff700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059] conn(0x7fcf8c0ac100 0x7fcf8c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:54.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.883+0000 7fcf8bfff700 1 -- 172.21.15.38:0/1112892940 --> [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcf94012ec0 con 0x7fcf8c0ac100 2024-03-26T06:35:54.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.883+0000 7fcf8bfff700 1 -- 172.21.15.38:0/1112892940 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf940cefc0 con 0x7fcfa4101740 2024-03-26T06:35:54.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.883+0000 7fcfa37fe700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059] conn(0x7fcf8c0ac100 0x7fcf8c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:54.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.883+0000 7fcfa27fc700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcf8c0a2920 0x7fcf8c0a4de0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009990 tx=0x7fcf98009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:54.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.883+0000 7fcfa37fe700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059] conn(0x7fcf8c0ac100 0x7fcf8c0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:54.888 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.884+0000 7fcf8bfff700 1 -- 172.21.15.38:0/1112892940 <== osd.6 v2:172.21.15.38:6818/757778059 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcf94012ec0 con 0x7fcf8c0ac100 2024-03-26T06:35:54.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.900+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1112892940 --> [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcfa41024f0 con 0x7fcf8c0ac100 2024-03-26T06:35:54.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.900+0000 7fcf8bfff700 1 -- 172.21.15.38:0/1112892940 <== osd.6 v2:172.21.15.38:6818/757778059 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcfa41024f0 con 0x7fcf8c0ac100 2024-03-26T06:35:54.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.900+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059] conn(0x7fcf8c0ac100 msgr2=0x7fcf8c0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:54.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.900+0000 7fcfa93fc700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059] conn(0x7fcf8c0ac100 0x7fcf8c0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:54.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.900+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcf8c0a2920 msgr2=0x7fcf8c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:54.903 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.901+0000 7fcfa93fc700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcf8c0a2920 0x7fcf8c0a4de0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcf98009990 tx=0x7fcf98009f10 comp rx=0 tx=0).stop 2024-03-26T06:35:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.901+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfa4101740 msgr2=0x7fcfa4113420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.901+0000 7fcfa93fc700 1 --2- 172.21.15.38:0/1112892940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcfa4101740 0x7fcfa4113420 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcf9400c920 tx=0x7fcf9400cce0 comp rx=0 tx=0).stop 2024-03-26T06:35:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.901+0000 7fcfa37fe700 1 -- 172.21.15.38:0/1112892940 reap_dead start 2024-03-26T06:35:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.901+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1112892940 shutdown_connections 2024-03-26T06:35:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.901+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1112892940 >> 172.21.15.38:0/1112892940 conn(0x7fcfa40772e0 msgr2=0x7fcfa41073e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.901+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1112892940 shutdown_connections 2024-03-26T06:35:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:54.901+0000 7fcfa93fc700 1 -- 172.21.15.38:0/1112892940 wait complete. 2024-03-26T06:35:55.490 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:55 smithi038 ceph-mon[32265]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T06:35:55.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:55 smithi170 ceph-mon[36912]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T06:35:55.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:55 smithi121 ceph-mon[36349]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T06:35:57.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:57 smithi170 ceph-mon[36912]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-26T06:35:57.737 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:57 smithi038 ceph-mon[32265]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-26T06:35:57.770 INFO:teuthology.orchestra.run.smithi038.stdout:201863462956 2024-03-26T06:35:57.770 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T06:35:57.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:57 smithi121 ceph-mon[36349]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-03-26T06:35:59.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.589+0000 7efd4a535700 1 -- 172.21.15.38:0/2260650613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd44107e10 msgr2=0x7efd441081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.589+0000 7efd4a535700 1 --2- 172.21.15.38:0/2260650613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd44107e10 0x7efd441081f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7efd340099c0 tx=0x7efd34009cd0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.590+0000 7efd4a535700 1 -- 172.21.15.38:0/2260650613 shutdown_connections 2024-03-26T06:35:59.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.590+0000 7efd4a535700 1 --2- 172.21.15.38:0/2260650613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4406aef0 0x7efd4410ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.590+0000 7efd4a535700 1 --2- 172.21.15.38:0/2260650613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd44107e10 0x7efd441081f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.590+0000 7efd4a535700 1 --2- 172.21.15.38:0/2260650613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd4406a530 0x7efd4406a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.590+0000 7efd4a535700 1 -- 172.21.15.38:0/2260650613 >> 172.21.15.38:0/2260650613 conn(0x7efd44077cf0 msgr2=0x7efd44078100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:59.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.590+0000 7efd4a535700 1 -- 172.21.15.38:0/2260650613 shutdown_connections 2024-03-26T06:35:59.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.590+0000 7efd4a535700 1 -- 172.21.15.38:0/2260650613 wait complete. 2024-03-26T06:35:59.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.590+0000 7efd4a535700 1 Processor -- start 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.590+0000 7efd4a535700 1 -- start start 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd4a535700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4406a530 0x7efd44100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd4a535700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd4406aef0 0x7efd441012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd4a535700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd44107e10 0x7efd44101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd4a535700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd44112090 con 0x7efd44107e10 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd4a535700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd44111f10 con 0x7efd4406aef0 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd4a535700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd44112210 con 0x7efd4406a530 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd48ad2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd44107e10 0x7efd44101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd48ad2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd44107e10 0x7efd44101810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47810/0 (socket says 172.21.15.38:47810) 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd48ad2700 1 -- 172.21.15.38:0/2684808007 learned_addr learned my addr 172.21.15.38:0/2684808007 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd43fff700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4406a530 0x7efd44100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:59.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd437fe700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd4406aef0 0x7efd441012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:59.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd48ad2700 1 -- 172.21.15.38:0/2684808007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4406a530 msgr2=0x7efd44100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd48ad2700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4406a530 0x7efd44100d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd48ad2700 1 -- 172.21.15.38:0/2684808007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd4406aef0 msgr2=0x7efd441012d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd48ad2700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd4406aef0 0x7efd441012d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd48ad2700 1 -- 172.21.15.38:0/2684808007 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd441a69f0 con 0x7efd44107e10 2024-03-26T06:35:59.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd43fff700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efd4406a530 0x7efd44100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:35:59.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd48ad2700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd44107e10 0x7efd44101810 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efd3c007d70 tx=0x7efd3c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:59.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.591+0000 7efd417fa700 1 -- 172.21.15.38:0/2684808007 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3c016020 con 0x7efd44107e10 2024-03-26T06:35:59.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.592+0000 7efd4a535700 1 -- 172.21.15.38:0/2684808007 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd441a6ce0 con 0x7efd44107e10 2024-03-26T06:35:59.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.592+0000 7efd417fa700 1 -- 172.21.15.38:0/2684808007 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd3c00ad40 con 0x7efd44107e10 2024-03-26T06:35:59.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.592+0000 7efd417fa700 1 -- 172.21.15.38:0/2684808007 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd3c011600 con 0x7efd44107e10 2024-03-26T06:35:59.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.592+0000 7efd4a535700 1 -- 172.21.15.38:0/2684808007 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd441a7220 con 0x7efd44107e10 2024-03-26T06:35:59.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.592+0000 7efd4a535700 1 -- 172.21.15.38:0/2684808007 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efd440624b0 con 0x7efd44107e10 2024-03-26T06:35:59.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.594+0000 7efd417fa700 1 -- 172.21.15.38:0/2684808007 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7efd3c019020 con 0x7efd44107e10 2024-03-26T06:35:59.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.595+0000 7efd417fa700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efd2c0a2920 0x7efd2c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:59.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.595+0000 7efd43fff700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efd2c0a2920 0x7efd2c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:59.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.595+0000 7efd417fa700 1 -- 172.21.15.38:0/2684808007 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efd3c0ced70 con 0x7efd44107e10 2024-03-26T06:35:59.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.595+0000 7efd417fa700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522] conn(0x7efd2c0ac100 0x7efd2c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:59.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.595+0000 7efd417fa700 1 -- 172.21.15.38:0/2684808007 --> [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efd3c021e50 con 0x7efd2c0ac100 2024-03-26T06:35:59.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.595+0000 7efd417fa700 1 -- 172.21.15.38:0/2684808007 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd3c0cf250 con 0x7efd44107e10 2024-03-26T06:35:59.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.595+0000 7efd43fff700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efd2c0a2920 0x7efd2c0a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efd34006010 tx=0x7efd34005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:59.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.595+0000 7efd437fe700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522] conn(0x7efd2c0ac100 0x7efd2c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:59.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.596+0000 7efd437fe700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522] conn(0x7efd2c0ac100 0x7efd2c0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:59.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.596+0000 7efd417fa700 1 -- 172.21.15.38:0/2684808007 <== osd.1 v2:172.21.15.121:6800/3506119522 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efd3c021e50 con 0x7efd2c0ac100 2024-03-26T06:35:59.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.613+0000 7efd4a535700 1 -- 172.21.15.38:0/2684808007 --> [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efd44062930 con 0x7efd2c0ac100 2024-03-26T06:35:59.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.613+0000 7efd417fa700 1 -- 172.21.15.38:0/2684808007 <== osd.1 v2:172.21.15.121:6800/3506119522 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efd44062930 con 0x7efd2c0ac100 2024-03-26T06:35:59.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.613+0000 7efd4a535700 1 -- 172.21.15.38:0/2684808007 >> [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522] conn(0x7efd2c0ac100 msgr2=0x7efd2c0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.613+0000 7efd4a535700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522] conn(0x7efd2c0ac100 0x7efd2c0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.614+0000 7efd4a535700 1 -- 172.21.15.38:0/2684808007 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efd2c0a2920 msgr2=0x7efd2c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.614+0000 7efd4a535700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efd2c0a2920 0x7efd2c0a4de0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7efd34006010 tx=0x7efd34005ab0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.614+0000 7efd4a535700 1 -- 172.21.15.38:0/2684808007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd44107e10 msgr2=0x7efd44101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.614+0000 7efd4a535700 1 --2- 172.21.15.38:0/2684808007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efd44107e10 0x7efd44101810 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efd3c007d70 tx=0x7efd3c00a340 comp rx=0 tx=0).stop 2024-03-26T06:35:59.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.614+0000 7efd48ad2700 1 -- 172.21.15.38:0/2684808007 reap_dead start 2024-03-26T06:35:59.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.614+0000 7efd4a535700 1 -- 172.21.15.38:0/2684808007 shutdown_connections 2024-03-26T06:35:59.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.614+0000 7efd4a535700 1 -- 172.21.15.38:0/2684808007 >> 172.21.15.38:0/2684808007 conn(0x7efd44077cf0 msgr2=0x7efd440ff7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:59.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.614+0000 7efd4a535700 1 -- 172.21.15.38:0/2684808007 shutdown_connections 2024-03-26T06:35:59.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.614+0000 7efd4a535700 1 -- 172.21.15.38:0/2684808007 wait complete. 2024-03-26T06:35:59.681 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:35:59 smithi038 ceph-mon[32265]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:35:59.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.781+0000 7f0230f46700 1 -- 172.21.15.38:0/3633346257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f022c101680 msgr2=0x7f022c10db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.781+0000 7f0230f46700 1 --2- 172.21.15.38:0/3633346257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f022c101680 0x7f022c10db60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f022400b0a0 tx=0x7f022400b3b0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.781+0000 7f0230f46700 1 -- 172.21.15.38:0/3633346257 shutdown_connections 2024-03-26T06:35:59.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.781+0000 7f0230f46700 1 --2- 172.21.15.38:0/3633346257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f022c101680 0x7f022c10db60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.781+0000 7f0230f46700 1 --2- 172.21.15.38:0/3633346257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022c106d20 0x7f022c107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.781+0000 7f0230f46700 1 --2- 172.21.15.38:0/3633346257 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f022c100ce0 0x7f022c101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.781+0000 7f0230f46700 1 -- 172.21.15.38:0/3633346257 >> 172.21.15.38:0/3633346257 conn(0x7f022c077c10 msgr2=0x7f022c078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.781+0000 7f0230f46700 1 -- 172.21.15.38:0/3633346257 shutdown_connections 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.781+0000 7f0230f46700 1 -- 172.21.15.38:0/3633346257 wait complete. 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f0230f46700 1 Processor -- start 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f0230f46700 1 -- start start 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f0230f46700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022c100ce0 0x7f022c198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f0230f46700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f022c101680 0x7f022c198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f0230f46700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f022c106d20 0x7f022c19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f0230f46700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c110f80 con 0x7f022c106d20 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f0230f46700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c110e00 con 0x7f022c100ce0 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f0230f46700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c111100 con 0x7f022c101680 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f022ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f022c106d20 0x7f022c19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f022ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f022c106d20 0x7f022c19cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47822/0 (socket says 172.21.15.38:47822) 2024-03-26T06:35:59.816 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f022ad9d700 1 -- 172.21.15.38:0/1096351725 learned_addr learned my addr 172.21.15.38:0/1096351725 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:35:59.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f022a59c700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022c100ce0 0x7f022c198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:59.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f0229d9b700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f022c101680 0x7f022c198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:59.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f022ad9d700 1 -- 172.21.15.38:0/1096351725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f022c101680 msgr2=0x7f022c198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.817 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f022ad9d700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f022c101680 0x7f022c198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f022ad9d700 1 -- 172.21.15.38:0/1096351725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022c100ce0 msgr2=0x7f022c198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f022ad9d700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022c100ce0 0x7f022c198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f022ad9d700 1 -- 172.21.15.38:0/1096351725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c19d4f0 con 0x7f022c106d20 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.782+0000 7f022a59c700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f022c100ce0 0x7f022c198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.783+0000 7f022ad9d700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f022c106d20 0x7f022c19cdb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f02240062a0 tx=0x7f0224004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.783+0000 7f021b7fe700 1 -- 172.21.15.38:0/1096351725 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0224021070 con 0x7f022c106d20 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.783+0000 7f0230f46700 1 -- 172.21.15.38:0/1096351725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022c19d780 con 0x7f022c106d20 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.783+0000 7f021b7fe700 1 -- 172.21.15.38:0/1096351725 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0224003c00 con 0x7f022c106d20 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.783+0000 7f021b7fe700 1 -- 172.21.15.38:0/1096351725 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0224004880 con 0x7f022c106d20 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.783+0000 7f0230f46700 1 -- 172.21.15.38:0/1096351725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022c1a5060 con 0x7f022c106d20 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.784+0000 7f0230f46700 1 -- 172.21.15.38:0/1096351725 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f022c104de0 con 0x7f022c106d20 2024-03-26T06:35:59.818 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.785+0000 7f021b7fe700 1 -- 172.21.15.38:0/1096351725 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0224007980 con 0x7f022c106d20 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.785+0000 7f021b7fe700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f02140a2920 0x7f02140a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.785+0000 7f022a59c700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f02140a2920 0x7f02140a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.786+0000 7f021b7fe700 1 -- 172.21.15.38:0/1096351725 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f02240cff40 con 0x7f022c106d20 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.786+0000 7f021b7fe700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971] conn(0x7f02140ac100 0x7f02140ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.786+0000 7f021b7fe700 1 -- 172.21.15.38:0/1096351725 --> [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02240020b0 con 0x7f02140ac100 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.786+0000 7f021b7fe700 1 -- 172.21.15.38:0/1096351725 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02240d02f0 con 0x7f022c106d20 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.786+0000 7f022a59c700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f02140a2920 0x7f02140a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f021c005950 tx=0x7f021c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.786+0000 7f0229d9b700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971] conn(0x7f02140ac100 0x7f02140ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.786+0000 7f0229d9b700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971] conn(0x7f02140ac100 0x7f02140ae520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.787+0000 7f021b7fe700 1 -- 172.21.15.38:0/1096351725 <== osd.8 v2:172.21.15.170:6816/1764725971 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f02240020b0 con 0x7f02140ac100 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.804+0000 7f0230f46700 1 -- 172.21.15.38:0/1096351725 --> [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f022c1023d0 con 0x7f02140ac100 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.804+0000 7f021b7fe700 1 -- 172.21.15.38:0/1096351725 <== osd.8 v2:172.21.15.170:6816/1764725971 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f022c1023d0 con 0x7f02140ac100 2024-03-26T06:35:59.819 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.804+0000 7f0230f46700 1 -- 172.21.15.38:0/1096351725 >> [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971] conn(0x7f02140ac100 msgr2=0x7f02140ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.804+0000 7f0230f46700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971] conn(0x7f02140ac100 0x7f02140ae520 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:35:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.804+0000 7f0230f46700 1 -- 172.21.15.38:0/1096351725 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f02140a2920 msgr2=0x7f02140a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.804+0000 7f0230f46700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f02140a2920 0x7f02140a4de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f021c005950 tx=0x7f021c00a300 comp rx=0 tx=0).stop 2024-03-26T06:35:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.804+0000 7f0230f46700 1 -- 172.21.15.38:0/1096351725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f022c106d20 msgr2=0x7f022c19cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:35:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.804+0000 7f0230f46700 1 --2- 172.21.15.38:0/1096351725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f022c106d20 0x7f022c19cdb0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f02240062a0 tx=0x7f0224004d70 comp rx=0 tx=0).stop 2024-03-26T06:35:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.805+0000 7f022ad9d700 1 -- 172.21.15.38:0/1096351725 reap_dead start 2024-03-26T06:35:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.805+0000 7f0230f46700 1 -- 172.21.15.38:0/1096351725 shutdown_connections 2024-03-26T06:35:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.805+0000 7f0230f46700 1 -- 172.21.15.38:0/1096351725 >> 172.21.15.38:0/1096351725 conn(0x7f022c077c10 msgr2=0x7f022c100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:35:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.805+0000 7f0230f46700 1 -- 172.21.15.38:0/1096351725 shutdown_connections 2024-03-26T06:35:59.820 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:35:59.805+0000 7f0230f46700 1 -- 172.21.15.38:0/1096351725 wait complete. 2024-03-26T06:35:59.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:35:59 smithi121 ceph-mon[36349]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:35:59.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:35:59 smithi170 ceph-mon[36912]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:36:01.477 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:01 smithi038 ceph-mon[32265]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:36:01.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:01 smithi121 ceph-mon[36349]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:36:01.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:01 smithi170 ceph-mon[36912]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:36:02.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.000+0000 7fd7d2d07700 1 -- 172.21.15.38:0/3548013480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc101920 msgr2=0x7fd7cc105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.000+0000 7fd7d2d07700 1 --2- 172.21.15.38:0/3548013480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc101920 0x7fd7cc105920 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8009990 tx=0x7fd7b8009ca0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.000+0000 7fd7d2d07700 1 -- 172.21.15.38:0/3548013480 shutdown_connections 2024-03-26T06:36:02.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.000+0000 7fd7d2d07700 1 --2- 172.21.15.38:0/3548013480 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7cc106000 0x7fd7cc10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.000+0000 7fd7d2d07700 1 --2- 172.21.15.38:0/3548013480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7cc100f70 0x7fd7cc101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.000+0000 7fd7d2d07700 1 --2- 172.21.15.38:0/3548013480 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc101920 0x7fd7cc105920 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.000+0000 7fd7d2d07700 1 -- 172.21.15.38:0/3548013480 >> 172.21.15.38:0/3548013480 conn(0x7fd7cc076d00 msgr2=0x7fd7cc077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:02.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7d2d07700 1 -- 172.21.15.38:0/3548013480 shutdown_connections 2024-03-26T06:36:02.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7d2d07700 1 -- 172.21.15.38:0/3548013480 wait complete. 2024-03-26T06:36:02.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7d2d07700 1 Processor -- start 2024-03-26T06:36:02.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7d2d07700 1 -- start start 2024-03-26T06:36:02.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7d2d07700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7cc100f70 0x7fd7cc1029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:02.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7d2d07700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc101920 0x7fd7cc102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:02.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7d2d07700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7cc106000 0x7fd7cc103440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:02.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7d2d07700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc110fe0 con 0x7fd7cc101920 2024-03-26T06:36:02.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7d2d07700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc110e60 con 0x7fd7cc106000 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7d2d07700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc111160 con 0x7fd7cc100f70 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7cbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc101920 0x7fd7cc102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7cbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc101920 0x7fd7cc102f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47852/0 (socket says 172.21.15.38:47852) 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.001+0000 7fd7cbfff700 1 -- 172.21.15.38:0/2816137151 learned_addr learned my addr 172.21.15.38:0/2816137151 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7d0aa3700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7cc100f70 0x7fd7cc1029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7d12a4700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7cc106000 0x7fd7cc103440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7cbfff700 1 -- 172.21.15.38:0/2816137151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7cc100f70 msgr2=0x7fd7cc1029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7cbfff700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd7cc100f70 0x7fd7cc1029c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7cbfff700 1 -- 172.21.15.38:0/2816137151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7cc106000 msgr2=0x7fd7cc103440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7cbfff700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7cc106000 0x7fd7cc103440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7cbfff700 1 -- 172.21.15.38:0/2816137151 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc1a6af0 con 0x7fd7cc101920 2024-03-26T06:36:02.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7cbfff700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc101920 0x7fd7cc102f00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8000c00 tx=0x7fd7b80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:02.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7c9ffb700 1 -- 172.21.15.38:0/2816137151 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8027070 con 0x7fd7cc101920 2024-03-26T06:36:02.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7d2d07700 1 -- 172.21.15.38:0/2816137151 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7cc1a6d80 con 0x7fd7cc101920 2024-03-26T06:36:02.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7c9ffb700 1 -- 172.21.15.38:0/2816137151 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7b8003e40 con 0x7fd7cc101920 2024-03-26T06:36:02.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7c9ffb700 1 -- 172.21.15.38:0/2816137151 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8022ce0 con 0x7fd7cc101920 2024-03-26T06:36:02.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.002+0000 7fd7d2d07700 1 -- 172.21.15.38:0/2816137151 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc1a7180 con 0x7fd7cc101920 2024-03-26T06:36:02.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.003+0000 7fd7d2d07700 1 -- 172.21.15.38:0/2816137151 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd7cc0624f0 con 0x7fd7cc101920 2024-03-26T06:36:02.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.004+0000 7fd7c9ffb700 1 -- 172.21.15.38:0/2816137151 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd7b802a070 con 0x7fd7cc101920 2024-03-26T06:36:02.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.005+0000 7fd7c9ffb700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd7bc0a2920 0x7fd7bc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:02.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.005+0000 7fd7d0aa3700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd7bc0a2920 0x7fd7bc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:02.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.005+0000 7fd7c9ffb700 1 -- 172.21.15.38:0/2816137151 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd7b80d01d0 con 0x7fd7cc101920 2024-03-26T06:36:02.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.005+0000 7fd7c9ffb700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891] conn(0x7fd7bc0ac100 0x7fd7bc0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:02.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.005+0000 7fd7c9ffb700 1 -- 172.21.15.38:0/2816137151 --> [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7b8010ec0 con 0x7fd7bc0ac100 2024-03-26T06:36:02.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.005+0000 7fd7c9ffb700 1 -- 172.21.15.38:0/2816137151 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7b80d06b0 con 0x7fd7cc101920 2024-03-26T06:36:02.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.005+0000 7fd7d0aa3700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd7bc0a2920 0x7fd7bc0a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0005950 tx=0x7fd7c000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:02.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.005+0000 7fd7d12a4700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891] conn(0x7fd7bc0ac100 0x7fd7bc0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:02.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.006+0000 7fd7d12a4700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891] conn(0x7fd7bc0ac100 0x7fd7bc0ae520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:02.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.007+0000 7fd7c9ffb700 1 -- 172.21.15.38:0/2816137151 <== osd.7 v2:172.21.15.121:6816/446835891 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd7b8010ec0 con 0x7fd7bc0ac100 2024-03-26T06:36:02.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.023+0000 7fd7d2d07700 1 -- 172.21.15.38:0/2816137151 --> [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd7cc062970 con 0x7fd7bc0ac100 2024-03-26T06:36:02.026 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7c9ffb700 1 -- 172.21.15.38:0/2816137151 <== osd.7 v2:172.21.15.121:6816/446835891 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd7cc062970 con 0x7fd7bc0ac100 2024-03-26T06:36:02.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7d2d07700 1 -- 172.21.15.38:0/2816137151 >> [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891] conn(0x7fd7bc0ac100 msgr2=0x7fd7bc0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7d2d07700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891] conn(0x7fd7bc0ac100 0x7fd7bc0ae520 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7d2d07700 1 -- 172.21.15.38:0/2816137151 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd7bc0a2920 msgr2=0x7fd7bc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7d2d07700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd7bc0a2920 0x7fd7bc0a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0005950 tx=0x7fd7c000a300 comp rx=0 tx=0).stop 2024-03-26T06:36:02.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7d2d07700 1 -- 172.21.15.38:0/2816137151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc101920 msgr2=0x7fd7cc102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.027 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7d2d07700 1 --2- 172.21.15.38:0/2816137151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc101920 0x7fd7cc102f00 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd7b8000c00 tx=0x7fd7b80047b0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7d12a4700 1 -- 172.21.15.38:0/2816137151 reap_dead start 2024-03-26T06:36:02.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7d2d07700 1 -- 172.21.15.38:0/2816137151 shutdown_connections 2024-03-26T06:36:02.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7d2d07700 1 -- 172.21.15.38:0/2816137151 >> 172.21.15.38:0/2816137151 conn(0x7fd7cc076d00 msgr2=0x7fd7cc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:02.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7d2d07700 1 -- 172.21.15.38:0/2816137151 shutdown_connections 2024-03-26T06:36:02.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:02.024+0000 7fd7d2d07700 1 -- 172.21.15.38:0/2816137151 wait complete. 2024-03-26T06:36:02.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.857+0000 7fd103dc1700 1 -- 172.21.15.38:0/1862210605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0fc106ed0 msgr2=0x7fd0fc1072b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.857+0000 7fd103dc1700 1 --2- 172.21.15.38:0/1862210605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0fc106ed0 0x7fd0fc1072b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0009990 tx=0x7fd0f0009ca0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.857+0000 7fd103dc1700 1 -- 172.21.15.38:0/1862210605 shutdown_connections 2024-03-26T06:36:02.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.857+0000 7fd103dc1700 1 --2- 172.21.15.38:0/1862210605 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc101980 0x7fd0fc10dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.857+0000 7fd103dc1700 1 --2- 172.21.15.38:0/1862210605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0fc106ed0 0x7fd0fc1072b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.857+0000 7fd103dc1700 1 --2- 172.21.15.38:0/1862210605 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0fc100fc0 0x7fd0fc101440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.857+0000 7fd103dc1700 1 -- 172.21.15.38:0/1862210605 >> 172.21.15.38:0/1862210605 conn(0x7fd0fc077b40 msgr2=0x7fd0fc077f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:02.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.857+0000 7fd103dc1700 1 -- 172.21.15.38:0/1862210605 shutdown_connections 2024-03-26T06:36:02.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.857+0000 7fd103dc1700 1 -- 172.21.15.38:0/1862210605 wait complete. 2024-03-26T06:36:02.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd103dc1700 1 Processor -- start 2024-03-26T06:36:02.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd103dc1700 1 -- start start 2024-03-26T06:36:02.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd103dc1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0fc100fc0 0x7fd0fc198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:02.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd103dc1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc101980 0x7fd0fc198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:02.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd103dc1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0fc106ed0 0x7fd0fc19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:02.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd103dc1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc111060 con 0x7fd0fc100fc0 2024-03-26T06:36:02.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd103dc1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc110ee0 con 0x7fd0fc101980 2024-03-26T06:36:02.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd103dc1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc1111e0 con 0x7fd0fc106ed0 2024-03-26T06:36:02.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd101b5d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0fc100fc0 0x7fd0fc198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:02.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd101b5d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0fc100fc0 0x7fd0fc198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47834/0 (socket says 172.21.15.38:47834) 2024-03-26T06:36:02.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd101b5d700 1 -- 172.21.15.38:0/3596653096 learned_addr learned my addr 172.21.15.38:0/3596653096 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd10135c700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc101980 0x7fd0fc198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd10235e700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0fc106ed0 0x7fd0fc19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd101b5d700 1 -- 172.21.15.38:0/3596653096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0fc106ed0 msgr2=0x7fd0fc19ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd101b5d700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0fc106ed0 0x7fd0fc19ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd101b5d700 1 -- 172.21.15.38:0/3596653096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc101980 msgr2=0x7fd0fc198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd101b5d700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc101980 0x7fd0fc198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd101b5d700 1 -- 172.21.15.38:0/3596653096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc19d5a0 con 0x7fd0fc100fc0 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd10235e700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd0fc106ed0 0x7fd0fc19ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.858+0000 7fd10135c700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0fc101980 0x7fd0fc198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.859+0000 7fd101b5d700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0fc100fc0 0x7fd0fc198590 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0000c00 tx=0x7fd0f00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.859+0000 7fd0eeffd700 1 -- 172.21.15.38:0/3596653096 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f0027070 con 0x7fd0fc100fc0 2024-03-26T06:36:02.580 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.859+0000 7fd103dc1700 1 -- 172.21.15.38:0/3596653096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc19d830 con 0x7fd0fc100fc0 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.859+0000 7fd0eeffd700 1 -- 172.21.15.38:0/3596653096 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0f0004050 con 0x7fd0fc100fc0 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.859+0000 7fd0eeffd700 1 -- 172.21.15.38:0/3596653096 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f000f460 con 0x7fd0fc100fc0 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.859+0000 7fd103dc1700 1 -- 172.21.15.38:0/3596653096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc1a5110 con 0x7fd0fc100fc0 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.860+0000 7fd103dc1700 1 -- 172.21.15.38:0/3596653096 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd0fc104df0 con 0x7fd0fc100fc0 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.861+0000 7fd0eeffd700 1 -- 172.21.15.38:0/3596653096 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd0f002a070 con 0x7fd0fc100fc0 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.861+0000 7fd0eeffd700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd0e80a2920 0x7fd0e80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.861+0000 7fd10135c700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd0e80a2920 0x7fd0e80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.862+0000 7fd0eeffd700 1 -- 172.21.15.38:0/3596653096 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd0f00d00c0 con 0x7fd0fc100fc0 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.862+0000 7fd0eeffd700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153] conn(0x7fd0e80ac100 0x7fd0e80ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.862+0000 7fd0eeffd700 1 -- 172.21.15.38:0/3596653096 --> [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd0f00043d0 con 0x7fd0e80ac100 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.862+0000 7fd0eeffd700 1 -- 172.21.15.38:0/3596653096 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd0f00d05a0 con 0x7fd0fc100fc0 2024-03-26T06:36:02.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.862+0000 7fd10135c700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd0e80a2920 0x7fd0e80a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0780b0 tx=0x7fd0f800b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:02.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.862+0000 7fd10235e700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153] conn(0x7fd0e80ac100 0x7fd0e80ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:02.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.862+0000 7fd10235e700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153] conn(0x7fd0e80ac100 0x7fd0e80ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:02.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.863+0000 7fd0eeffd700 1 -- 172.21.15.38:0/3596653096 <== osd.5 v2:172.21.15.170:6808/718927153 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd0f00043d0 con 0x7fd0e80ac100 2024-03-26T06:36:02.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.880+0000 7fd103dc1700 1 -- 172.21.15.38:0/3596653096 --> [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd0fc102460 con 0x7fd0e80ac100 2024-03-26T06:36:02.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.880+0000 7fd0eeffd700 1 -- 172.21.15.38:0/3596653096 <== osd.5 v2:172.21.15.170:6808/718927153 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd0fc102460 con 0x7fd0e80ac100 2024-03-26T06:36:02.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.880+0000 7fd103dc1700 1 -- 172.21.15.38:0/3596653096 >> [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153] conn(0x7fd0e80ac100 msgr2=0x7fd0e80ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.880+0000 7fd103dc1700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153] conn(0x7fd0e80ac100 0x7fd0e80ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.881+0000 7fd103dc1700 1 -- 172.21.15.38:0/3596653096 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd0e80a2920 msgr2=0x7fd0e80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.881+0000 7fd103dc1700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd0e80a2920 0x7fd0e80a4de0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0780b0 tx=0x7fd0f800b2c0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.881+0000 7fd103dc1700 1 -- 172.21.15.38:0/3596653096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0fc100fc0 msgr2=0x7fd0fc198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:02.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.881+0000 7fd103dc1700 1 --2- 172.21.15.38:0/3596653096 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd0fc100fc0 0x7fd0fc198590 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd0f0000c00 tx=0x7fd0f00047b0 comp rx=0 tx=0).stop 2024-03-26T06:36:02.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.881+0000 7fd10235e700 1 -- 172.21.15.38:0/3596653096 reap_dead start 2024-03-26T06:36:02.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.881+0000 7fd103dc1700 1 -- 172.21.15.38:0/3596653096 shutdown_connections 2024-03-26T06:36:02.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.881+0000 7fd103dc1700 1 -- 172.21.15.38:0/3596653096 >> 172.21.15.38:0/3596653096 conn(0x7fd0fc077b40 msgr2=0x7fd0fc100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:02.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.881+0000 7fd103dc1700 1 -- 172.21.15.38:0/3596653096 shutdown_connections 2024-03-26T06:36:02.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:00.881+0000 7fd103dc1700 1 -- 172.21.15.38:0/3596653096 wait complete. 2024-03-26T06:36:03.272 INFO:teuthology.orchestra.run.smithi038.stdout:167503724594 2024-03-26T06:36:03.273 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T06:36:03.508 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:03 smithi038 ceph-mon[32265]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:36:03.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.608+0000 7fdf1dc13700 1 -- 172.21.15.38:0/184751429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf18107e30 msgr2=0x7fdf18108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:03.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.608+0000 7fdf1dc13700 1 --2- 172.21.15.38:0/184751429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf18107e30 0x7fdf18108210 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fdf080099c0 tx=0x7fdf08009cd0 comp rx=0 tx=0).stop 2024-03-26T06:36:03.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.608+0000 7fdf1dc13700 1 -- 172.21.15.38:0/184751429 shutdown_connections 2024-03-26T06:36:03.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.608+0000 7fdf1dc13700 1 --2- 172.21.15.38:0/184751429 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf1806a430 0x7fdf1810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:03.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.608+0000 7fdf1dc13700 1 --2- 172.21.15.38:0/184751429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf18107e30 0x7fdf18108210 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:03.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.608+0000 7fdf1dc13700 1 --2- 172.21.15.38:0/184751429 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf18069a90 0x7fdf18069ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:03.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.608+0000 7fdf1dc13700 1 -- 172.21.15.38:0/184751429 >> 172.21.15.38:0/184751429 conn(0x7fdf18077c10 msgr2=0x7fdf18078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:03.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.608+0000 7fdf1dc13700 1 -- 172.21.15.38:0/184751429 shutdown_connections 2024-03-26T06:36:03.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.608+0000 7fdf1dc13700 1 -- 172.21.15.38:0/184751429 wait complete. 2024-03-26T06:36:03.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.608+0000 7fdf1dc13700 1 Processor -- start 2024-03-26T06:36:03.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.608+0000 7fdf1dc13700 1 -- start start 2024-03-26T06:36:03.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf1dc13700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf18069a90 0x7fdf181985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:03.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf1dc13700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf1806a430 0x7fdf18198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:03.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf1dc13700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf18107e30 0x7fdf1819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:03.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf1dc13700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf18112070 con 0x7fdf18069a90 2024-03-26T06:36:03.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf1dc13700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf18111ef0 con 0x7fdf18107e30 2024-03-26T06:36:03.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf1dc13700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf181121f0 con 0x7fdf1806a430 2024-03-26T06:36:03.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf177fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf18069a90 0x7fdf181985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:03.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf177fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf18069a90 0x7fdf181985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47872/0 (socket says 172.21.15.38:47872) 2024-03-26T06:36:03.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf177fe700 1 -- 172.21.15.38:0/2652335385 learned_addr learned my addr 172.21.15.38:0/2652335385 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:03.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf16ffd700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf1806a430 0x7fdf18198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:03.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf17fff700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf18107e30 0x7fdf1819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:03.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf177fe700 1 -- 172.21.15.38:0/2652335385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf1806a430 msgr2=0x7fdf18198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:03.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf177fe700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdf1806a430 0x7fdf18198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:03.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf177fe700 1 -- 172.21.15.38:0/2652335385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf18107e30 msgr2=0x7fdf1819ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:03.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf177fe700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf18107e30 0x7fdf1819ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:03.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf177fe700 1 -- 172.21.15.38:0/2652335385 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1819d560 con 0x7fdf18069a90 2024-03-26T06:36:03.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf177fe700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf18069a90 0x7fdf181985a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdf08009fd0 tx=0x7fdf08003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:03.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf14ff9700 1 -- 172.21.15.38:0/2652335385 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf08027070 con 0x7fdf18069a90 2024-03-26T06:36:03.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf1dc13700 1 -- 172.21.15.38:0/2652335385 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1819d7f0 con 0x7fdf18069a90 2024-03-26T06:36:03.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf14ff9700 1 -- 172.21.15.38:0/2652335385 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf08004150 con 0x7fdf18069a90 2024-03-26T06:36:03.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.609+0000 7fdf14ff9700 1 -- 172.21.15.38:0/2652335385 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0800f460 con 0x7fdf18069a90 2024-03-26T06:36:03.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.610+0000 7fdf1dc13700 1 -- 172.21.15.38:0/2652335385 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf181a50d0 con 0x7fdf18069a90 2024-03-26T06:36:03.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.610+0000 7fdf1dc13700 1 -- 172.21.15.38:0/2652335385 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdf180624f0 con 0x7fdf18069a90 2024-03-26T06:36:03.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.612+0000 7fdf14ff9700 1 -- 172.21.15.38:0/2652335385 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fdf0802a070 con 0x7fdf18069a90 2024-03-26T06:36:03.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.612+0000 7fdf14ff9700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdf000a2920 0x7fdf000a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:03.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.612+0000 7fdf14ff9700 1 -- 172.21.15.38:0/2652335385 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fdf080d0410 con 0x7fdf18069a90 2024-03-26T06:36:03.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.612+0000 7fdf16ffd700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdf000a2920 0x7fdf000a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:03.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.612+0000 7fdf14ff9700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291] conn(0x7fdf000ac100 0x7fdf000ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:03.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.613+0000 7fdf14ff9700 1 -- 172.21.15.38:0/2652335385 --> [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf080043d0 con 0x7fdf000ac100 2024-03-26T06:36:03.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.613+0000 7fdf14ff9700 1 -- 172.21.15.38:0/2652335385 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf080d08f0 con 0x7fdf18069a90 2024-03-26T06:36:03.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.613+0000 7fdf17fff700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291] conn(0x7fdf000ac100 0x7fdf000ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:03.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.613+0000 7fdf16ffd700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdf000a2920 0x7fdf000a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c005fd0 tx=0x7fdf0c00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:03.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.613+0000 7fdf17fff700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291] conn(0x7fdf000ac100 0x7fdf000ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:03.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.614+0000 7fdf14ff9700 1 -- 172.21.15.38:0/2652335385 <== osd.3 v2:172.21.15.38:6810/3343509291 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fdf080043d0 con 0x7fdf000ac100 2024-03-26T06:36:03.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.632+0000 7fdf1dc13700 1 -- 172.21.15.38:0/2652335385 --> [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdf180628f0 con 0x7fdf000ac100 2024-03-26T06:36:03.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.632+0000 7fdf14ff9700 1 -- 172.21.15.38:0/2652335385 <== osd.3 v2:172.21.15.38:6810/3343509291 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fdf180628f0 con 0x7fdf000ac100 2024-03-26T06:36:03.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.633+0000 7fdf1dc13700 1 -- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291] conn(0x7fdf000ac100 msgr2=0x7fdf000ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:03.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.633+0000 7fdf1dc13700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291] conn(0x7fdf000ac100 0x7fdf000ae520 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:03.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.633+0000 7fdf1dc13700 1 -- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdf000a2920 msgr2=0x7fdf000a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:03.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.633+0000 7fdf1dc13700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdf000a2920 0x7fdf000a4de0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c005fd0 tx=0x7fdf0c00a560 comp rx=0 tx=0).stop 2024-03-26T06:36:03.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.633+0000 7fdf1dc13700 1 -- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf18069a90 msgr2=0x7fdf181985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:03.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.633+0000 7fdf1dc13700 1 --2- 172.21.15.38:0/2652335385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf18069a90 0x7fdf181985a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fdf08009fd0 tx=0x7fdf08003730 comp rx=0 tx=0).stop 2024-03-26T06:36:03.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.633+0000 7fdf17fff700 1 -- 172.21.15.38:0/2652335385 reap_dead start 2024-03-26T06:36:03.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.633+0000 7fdf1dc13700 1 -- 172.21.15.38:0/2652335385 shutdown_connections 2024-03-26T06:36:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.633+0000 7fdf1dc13700 1 -- 172.21.15.38:0/2652335385 >> 172.21.15.38:0/2652335385 conn(0x7fdf18077c10 msgr2=0x7fdf180ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.633+0000 7fdf1dc13700 1 -- 172.21.15.38:0/2652335385 shutdown_connections 2024-03-26T06:36:03.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:03.633+0000 7fdf1dc13700 1 -- 172.21.15.38:0/2652335385 wait complete. 2024-03-26T06:36:03.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:03 smithi121 ceph-mon[36349]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:36:03.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:03 smithi170 ceph-mon[36912]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:36:04.651 INFO:teuthology.orchestra.run.smithi038.stdout:55834574918 2024-03-26T06:36:04.652 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T06:36:05.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.526+0000 7f6a32c54700 1 -- 172.21.15.38:0/4188193360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a2c107e60 msgr2=0x7f6a2c108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.526+0000 7f6a32c54700 1 --2- 172.21.15.38:0/4188193360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a2c107e60 0x7f6a2c108240 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6a200099c0 tx=0x7f6a20009cd0 comp rx=0 tx=0).stop 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.526+0000 7f6a32c54700 1 -- 172.21.15.38:0/4188193360 shutdown_connections 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.526+0000 7f6a32c54700 1 --2- 172.21.15.38:0/4188193360 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a2c06ae80 0x7f6a2c10eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.526+0000 7f6a32c54700 1 --2- 172.21.15.38:0/4188193360 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a2c107e60 0x7f6a2c108240 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.526+0000 7f6a32c54700 1 --2- 172.21.15.38:0/4188193360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a2c06a4e0 0x7f6a2c06a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.526+0000 7f6a32c54700 1 -- 172.21.15.38:0/4188193360 >> 172.21.15.38:0/4188193360 conn(0x7f6a2c076d00 msgr2=0x7f6a2c077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.526+0000 7f6a32c54700 1 -- 172.21.15.38:0/4188193360 shutdown_connections 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.526+0000 7f6a32c54700 1 -- 172.21.15.38:0/4188193360 wait complete. 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a32c54700 1 Processor -- start 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a32c54700 1 -- start start 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a32c54700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a2c06a4e0 0x7f6a2c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a32c54700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a2c06ae80 0x7f6a2c198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:05.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a32c54700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a2c107e60 0x7f6a2c19cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a32c54700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c1120a0 con 0x7f6a2c107e60 2024-03-26T06:36:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a32c54700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c111f20 con 0x7f6a2c06a4e0 2024-03-26T06:36:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a32c54700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c112220 con 0x7f6a2c06ae80 2024-03-26T06:36:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a311f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a2c107e60 0x7f6a2c19cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a311f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a2c107e60 0x7f6a2c19cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42468/0 (socket says 172.21.15.38:42468) 2024-03-26T06:36:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a311f1700 1 -- 172.21.15.38:0/4146968790 learned_addr learned my addr 172.21.15.38:0/4146968790 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a309f0700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a2c06a4e0 0x7f6a2c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:05.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.527+0000 7f6a2bfff700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a2c06ae80 0x7f6a2c198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.528+0000 7f6a311f1700 1 -- 172.21.15.38:0/4146968790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a2c06ae80 msgr2=0x7f6a2c198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.528+0000 7f6a311f1700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6a2c06ae80 0x7f6a2c198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.528+0000 7f6a311f1700 1 -- 172.21.15.38:0/4146968790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a2c06a4e0 msgr2=0x7f6a2c1984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.528+0000 7f6a311f1700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a2c06a4e0 0x7f6a2c1984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.528+0000 7f6a311f1700 1 -- 172.21.15.38:0/4146968790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2c19d500 con 0x7f6a2c107e60 2024-03-26T06:36:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.528+0000 7f6a311f1700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a2c107e60 0x7f6a2c19cdc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6a24007ee0 tx=0x7f6a2400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.528+0000 7f6a29ffb700 1 -- 172.21.15.38:0/4146968790 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a24004bb0 con 0x7f6a2c107e60 2024-03-26T06:36:05.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.528+0000 7f6a29ffb700 1 -- 172.21.15.38:0/4146968790 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a24004d50 con 0x7f6a2c107e60 2024-03-26T06:36:05.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.528+0000 7f6a29ffb700 1 -- 172.21.15.38:0/4146968790 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a24005960 con 0x7f6a2c107e60 2024-03-26T06:36:05.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.528+0000 7f6a32c54700 1 -- 172.21.15.38:0/4146968790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2c19d7f0 con 0x7f6a2c107e60 2024-03-26T06:36:05.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.528+0000 7f6a32c54700 1 -- 172.21.15.38:0/4146968790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2c1a5070 con 0x7f6a2c107e60 2024-03-26T06:36:05.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.529+0000 7f6a32c54700 1 -- 172.21.15.38:0/4146968790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6a2c062570 con 0x7f6a2c107e60 2024-03-26T06:36:05.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.530+0000 7f6a29ffb700 1 -- 172.21.15.38:0/4146968790 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f6a2401a020 con 0x7f6a2c107e60 2024-03-26T06:36:05.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.531+0000 7f6a29ffb700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6a1c0a29a0 0x7f6a1c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:05.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.531+0000 7f6a309f0700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6a1c0a29a0 0x7f6a1c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:05.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.531+0000 7f6a29ffb700 1 -- 172.21.15.38:0/4146968790 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f6a240ce840 con 0x7f6a2c107e60 2024-03-26T06:36:05.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.531+0000 7f6a29ffb700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510] conn(0x7f6a1c0ac180 0x7f6a1c0ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:05.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.531+0000 7f6a29ffb700 1 -- 172.21.15.38:0/4146968790 --> [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a2400be50 con 0x7f6a1c0ac180 2024-03-26T06:36:05.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.531+0000 7f6a29ffb700 1 -- 172.21.15.38:0/4146968790 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a240cebf0 con 0x7f6a2c107e60 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.531+0000 7f6a2bfff700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510] conn(0x7f6a1c0ac180 0x7f6a1c0ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.531+0000 7f6a309f0700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6a1c0a29a0 0x7f6a1c0a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6a20000c00 tx=0x7f6a20005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.532+0000 7f6a2bfff700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510] conn(0x7f6a1c0ac180 0x7f6a1c0ae5a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.533+0000 7f6a29ffb700 1 -- 172.21.15.38:0/4146968790 <== osd.9 v2:172.21.15.38:6826/539511510 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f6a2400be50 con 0x7f6a1c0ac180 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.549+0000 7f6a32c54700 1 -- 172.21.15.38:0/4146968790 --> [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6a2c0629f0 con 0x7f6a1c0ac180 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.550+0000 7f6a29ffb700 1 -- 172.21.15.38:0/4146968790 <== osd.9 v2:172.21.15.38:6826/539511510 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6a2c0629f0 con 0x7f6a1c0ac180 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.550+0000 7f6a32c54700 1 -- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510] conn(0x7f6a1c0ac180 msgr2=0x7f6a1c0ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.550+0000 7f6a32c54700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510] conn(0x7f6a1c0ac180 0x7f6a1c0ae5a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.550+0000 7f6a32c54700 1 -- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6a1c0a29a0 msgr2=0x7f6a1c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.550+0000 7f6a32c54700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6a1c0a29a0 0x7f6a1c0a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6a20000c00 tx=0x7f6a20005fb0 comp rx=0 tx=0).stop 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.550+0000 7f6a32c54700 1 -- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a2c107e60 msgr2=0x7f6a2c19cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.550+0000 7f6a32c54700 1 --2- 172.21.15.38:0/4146968790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a2c107e60 0x7f6a2c19cdc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6a24007ee0 tx=0x7f6a2400d3b0 comp rx=0 tx=0).stop 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.550+0000 7f6a311f1700 1 -- 172.21.15.38:0/4146968790 reap_dead start 2024-03-26T06:36:05.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.550+0000 7f6a32c54700 1 -- 172.21.15.38:0/4146968790 shutdown_connections 2024-03-26T06:36:05.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.550+0000 7f6a32c54700 1 -- 172.21.15.38:0/4146968790 >> 172.21.15.38:0/4146968790 conn(0x7f6a2c076d00 msgr2=0x7f6a2c0ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:05.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.551+0000 7f6a32c54700 1 -- 172.21.15.38:0/4146968790 shutdown_connections 2024-03-26T06:36:05.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:05.551+0000 7f6a32c54700 1 -- 172.21.15.38:0/4146968790 wait complete. 2024-03-26T06:36:05.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:05 smithi038 ceph-mon[32265]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:36:05.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:05 smithi121 ceph-mon[36349]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:36:05.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:05 smithi170 ceph-mon[36912]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:36:06.388 INFO:teuthology.orchestra.run.smithi038.stdout:223338299432 2024-03-26T06:36:06.388 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T06:36:06.924 INFO:teuthology.orchestra.run.smithi038.stdout:111669149757 2024-03-26T06:36:06.924 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T06:36:07.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:07 smithi038 ceph-mon[32265]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:36:07.720 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:07 smithi170 ceph-mon[36912]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:36:07.747 INFO:teuthology.orchestra.run.smithi038.stdout:257698037795 2024-03-26T06:36:07.747 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-26T06:36:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:07 smithi121 ceph-mon[36349]: pgmap v306: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:36:08.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.559+0000 7f1281a75700 1 -- 172.21.15.38:0/4089816270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f127c101e30 msgr2=0x7f127c1022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.559+0000 7f1281a75700 1 --2- 172.21.15.38:0/4089816270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f127c101e30 0x7f127c1022b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f12700099c0 tx=0x7f1270009cd0 comp rx=0 tx=0).stop 2024-03-26T06:36:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.559+0000 7f1281a75700 1 -- 172.21.15.38:0/4089816270 shutdown_connections 2024-03-26T06:36:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.559+0000 7f1281a75700 1 --2- 172.21.15.38:0/4089816270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c1027f0 0x7f127c10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.559+0000 7f1281a75700 1 --2- 172.21.15.38:0/4089816270 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f127c107e10 0x7f127c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.559+0000 7f1281a75700 1 --2- 172.21.15.38:0/4089816270 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f127c101e30 0x7f127c1022b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.559+0000 7f1281a75700 1 -- 172.21.15.38:0/4089816270 >> 172.21.15.38:0/4089816270 conn(0x7f127c078f10 msgr2=0x7f127c079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.559+0000 7f1281a75700 1 -- 172.21.15.38:0/4089816270 shutdown_connections 2024-03-26T06:36:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.559+0000 7f1281a75700 1 -- 172.21.15.38:0/4089816270 wait complete. 2024-03-26T06:36:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.560+0000 7f1281a75700 1 Processor -- start 2024-03-26T06:36:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.560+0000 7f1281a75700 1 -- start start 2024-03-26T06:36:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.560+0000 7f1281a75700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f127c101e30 0x7f127c198480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.560+0000 7f1281a75700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c1027f0 0x7f127c1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.560+0000 7f1281a75700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f127c107e10 0x7f127c19cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.560+0000 7f1281a75700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c112110 con 0x7f127c101e30 2024-03-26T06:36:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.560+0000 7f1281a75700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c111f90 con 0x7f127c1027f0 2024-03-26T06:36:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.560+0000 7f1281a75700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127c112290 con 0x7f127c107e10 2024-03-26T06:36:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.560+0000 7f127b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f127c101e30 0x7f127c198480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f127b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f127c101e30 0x7f127c198480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42490/0 (socket says 172.21.15.38:42490) 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f127b7fe700 1 -- 172.21.15.38:0/311434610 learned_addr learned my addr 172.21.15.38:0/311434610 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f127affd700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c1027f0 0x7f127c1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f127bfff700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f127c107e10 0x7f127c19cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f127b7fe700 1 -- 172.21.15.38:0/311434610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f127c107e10 msgr2=0x7f127c19cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f127b7fe700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f127c107e10 0x7f127c19cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f127b7fe700 1 -- 172.21.15.38:0/311434610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c1027f0 msgr2=0x7f127c1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f127b7fe700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c1027f0 0x7f127c1989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f127b7fe700 1 -- 172.21.15.38:0/311434610 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127c19d490 con 0x7f127c101e30 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f127affd700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f127c1027f0 0x7f127c1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f127b7fe700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f127c101e30 0x7f127c198480 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f126c00b530 tx=0x7f126c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:08.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f1278ff9700 1 -- 172.21.15.38:0/311434610 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c015020 con 0x7f127c101e30 2024-03-26T06:36:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f1278ff9700 1 -- 172.21.15.38:0/311434610 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f126c011d10 con 0x7f127c101e30 2024-03-26T06:36:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f1281a75700 1 -- 172.21.15.38:0/311434610 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f127c19d780 con 0x7f127c101e30 2024-03-26T06:36:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f1278ff9700 1 -- 172.21.15.38:0/311434610 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c010e50 con 0x7f127c101e30 2024-03-26T06:36:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.561+0000 7f1281a75700 1 -- 172.21.15.38:0/311434610 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f127c1a5000 con 0x7f127c101e30 2024-03-26T06:36:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.562+0000 7f1281a75700 1 -- 172.21.15.38:0/311434610 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f127c101300 con 0x7f127c101e30 2024-03-26T06:36:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.563+0000 7f1278ff9700 1 -- 172.21.15.38:0/311434610 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f126c018020 con 0x7f127c101e30 2024-03-26T06:36:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.564+0000 7f1278ff9700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f12640a2b40 0x7f12640a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.564+0000 7f127affd700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f12640a2b40 0x7f12640a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.564+0000 7f1278ff9700 1 -- 172.21.15.38:0/311434610 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f126c0ce8c0 con 0x7f127c101e30 2024-03-26T06:36:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.564+0000 7f1278ff9700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754] conn(0x7f12640ac320 0x7f12640ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.564+0000 7f1278ff9700 1 -- 172.21.15.38:0/311434610 --> [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f126c019e70 con 0x7f12640ac320 2024-03-26T06:36:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.564+0000 7f1278ff9700 1 -- 172.21.15.38:0/311434610 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f126c0cec70 con 0x7f127c101e30 2024-03-26T06:36:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.564+0000 7f127affd700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f12640a2b40 0x7f12640a5000 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f127c0794a0 tx=0x7f1270005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.564+0000 7f127bfff700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754] conn(0x7f12640ac320 0x7f12640ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.565+0000 7f127bfff700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754] conn(0x7f12640ac320 0x7f12640ae740 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.566+0000 7f1278ff9700 1 -- 172.21.15.38:0/311434610 <== osd.11 v2:172.21.15.170:6824/3915107754 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f126c019e70 con 0x7f12640ac320 2024-03-26T06:36:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.582+0000 7f1281a75700 1 -- 172.21.15.38:0/311434610 --> [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f127c1034c0 con 0x7f12640ac320 2024-03-26T06:36:08.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f1278ff9700 1 -- 172.21.15.38:0/311434610 <== osd.11 v2:172.21.15.170:6824/3915107754 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f127c1034c0 con 0x7f12640ac320 2024-03-26T06:36:08.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f1281a75700 1 -- 172.21.15.38:0/311434610 >> [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754] conn(0x7f12640ac320 msgr2=0x7f12640ae740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f1281a75700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754] conn(0x7f12640ac320 0x7f12640ae740 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f1281a75700 1 -- 172.21.15.38:0/311434610 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f12640a2b40 msgr2=0x7f12640a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f1281a75700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f12640a2b40 0x7f12640a5000 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f127c0794a0 tx=0x7f1270005fd0 comp rx=0 tx=0).stop 2024-03-26T06:36:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f1281a75700 1 -- 172.21.15.38:0/311434610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f127c101e30 msgr2=0x7f127c198480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f1281a75700 1 --2- 172.21.15.38:0/311434610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f127c101e30 0x7f127c198480 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f126c00b530 tx=0x7f126c00b8f0 comp rx=0 tx=0).stop 2024-03-26T06:36:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f127bfff700 1 -- 172.21.15.38:0/311434610 reap_dead start 2024-03-26T06:36:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f1281a75700 1 -- 172.21.15.38:0/311434610 shutdown_connections 2024-03-26T06:36:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f1281a75700 1 -- 172.21.15.38:0/311434610 >> 172.21.15.38:0/311434610 conn(0x7f127c078f10 msgr2=0x7f127c1008a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f1281a75700 1 -- 172.21.15.38:0/311434610 shutdown_connections 2024-03-26T06:36:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:08.583+0000 7f1281a75700 1 -- 172.21.15.38:0/311434610 wait complete. 2024-03-26T06:36:09.243 INFO:teuthology.orchestra.run.smithi038.stdout:347892350998 2024-03-26T06:36:09.243 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-26T06:36:09.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:09 smithi038 ceph-mon[32265]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:09.699 INFO:teuthology.orchestra.run.smithi038.stdout:292057776158 2024-03-26T06:36:09.700 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-26T06:36:09.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:09 smithi121 ceph-mon[36349]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:09.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:09 smithi170 ceph-mon[36912]: pgmap v307: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:10.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.208+0000 7f4110237700 1 -- 172.21.15.38:0/2872006195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f410806a430 msgr2=0x7f410810ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.208+0000 7f4110237700 1 --2- 172.21.15.38:0/2872006195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f410806a430 0x7f410810ec70 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f41000099c0 tx=0x7f4100009cd0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.208+0000 7f4110237700 1 -- 172.21.15.38:0/2872006195 shutdown_connections 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.208+0000 7f4110237700 1 --2- 172.21.15.38:0/2872006195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f410806a430 0x7f410810ec70 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.208+0000 7f4110237700 1 --2- 172.21.15.38:0/2872006195 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4108107e30 0x7f4108108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.208+0000 7f4110237700 1 --2- 172.21.15.38:0/2872006195 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4108069a90 0x7f4108069ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.208+0000 7f4110237700 1 -- 172.21.15.38:0/2872006195 >> 172.21.15.38:0/2872006195 conn(0x7f4108077c10 msgr2=0x7f4108078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.208+0000 7f4110237700 1 -- 172.21.15.38:0/2872006195 shutdown_connections 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.208+0000 7f4110237700 1 -- 172.21.15.38:0/2872006195 wait complete. 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.208+0000 7f4110237700 1 Processor -- start 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f4110237700 1 -- start start 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f4110237700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4108069a90 0x7f4108198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f4110237700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f410806a430 0x7f4108198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f4110237700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4108107e30 0x7f410819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:10.213 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f4110237700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4108112180 con 0x7f4108069a90 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f4110237700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4108112000 con 0x7f410806a430 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f4110237700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4108112300 con 0x7f4108107e30 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410dfd3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4108069a90 0x7f4108198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410dfd3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4108069a90 0x7f4108198590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42508/0 (socket says 172.21.15.38:42508) 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410dfd3700 1 -- 172.21.15.38:0/4262812160 learned_addr learned my addr 172.21.15.38:0/4262812160 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410d7d2700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f410806a430 0x7f4108198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410e7d4700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4108107e30 0x7f410819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410dfd3700 1 -- 172.21.15.38:0/4262812160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4108107e30 msgr2=0x7f410819ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410dfd3700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4108107e30 0x7f410819ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410dfd3700 1 -- 172.21.15.38:0/4262812160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f410806a430 msgr2=0x7f4108198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410dfd3700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f410806a430 0x7f4108198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410dfd3700 1 -- 172.21.15.38:0/4262812160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410819d550 con 0x7f4108069a90 2024-03-26T06:36:10.214 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410e7d4700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4108107e30 0x7f410819ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.209+0000 7f410dfd3700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4108069a90 0x7f4108198590 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f40f80060b0 tx=0x7f40f800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.210+0000 7f4106ffd700 1 -- 172.21.15.38:0/4262812160 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f801a070 con 0x7f4108069a90 2024-03-26T06:36:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.210+0000 7f4106ffd700 1 -- 172.21.15.38:0/4262812160 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40f800ddd0 con 0x7f4108069a90 2024-03-26T06:36:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.210+0000 7f4110237700 1 -- 172.21.15.38:0/4262812160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410819d840 con 0x7f4108069a90 2024-03-26T06:36:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.210+0000 7f4106ffd700 1 -- 172.21.15.38:0/4262812160 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f80147a0 con 0x7f4108069a90 2024-03-26T06:36:10.215 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.210+0000 7f4110237700 1 -- 172.21.15.38:0/4262812160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41081a50c0 con 0x7f4108069a90 2024-03-26T06:36:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.210+0000 7f4110237700 1 -- 172.21.15.38:0/4262812160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4108062570 con 0x7f4108069a90 2024-03-26T06:36:10.216 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.212+0000 7f4106ffd700 1 -- 172.21.15.38:0/4262812160 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f40f801d070 con 0x7f4108069a90 2024-03-26T06:36:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.212+0000 7f4106ffd700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f40f00a29a0 0x7f40f00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.212+0000 7f410d7d2700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f40f00a29a0 0x7f40f00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.212+0000 7f4106ffd700 1 -- 172.21.15.38:0/4262812160 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f40f80ceab0 con 0x7f4108069a90 2024-03-26T06:36:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.213+0000 7f4106ffd700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871] conn(0x7f40f00ac180 0x7f40f00ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.213+0000 7f4106ffd700 1 -- 172.21.15.38:0/4262812160 --> [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40f8017ec0 con 0x7f40f00ac180 2024-03-26T06:36:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.213+0000 7f4106ffd700 1 -- 172.21.15.38:0/4262812160 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40f80cef90 con 0x7f4108069a90 2024-03-26T06:36:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.213+0000 7f410d7d2700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f40f00a29a0 0x7f40f00a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4108078180 tx=0x7f40fc009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.213+0000 7f410e7d4700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871] conn(0x7f40f00ac180 0x7f40f00ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.213+0000 7f410e7d4700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871] conn(0x7f40f00ac180 0x7f40f00ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:10.217 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.214+0000 7f4106ffd700 1 -- 172.21.15.38:0/4262812160 <== osd.2 v2:172.21.15.170:6800/1189530871 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f40f8017ec0 con 0x7f40f00ac180 2024-03-26T06:36:10.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.231+0000 7f4110237700 1 -- 172.21.15.38:0/4262812160 --> [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f41080629a0 con 0x7f40f00ac180 2024-03-26T06:36:10.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.231+0000 7f4106ffd700 1 -- 172.21.15.38:0/4262812160 <== osd.2 v2:172.21.15.170:6800/1189530871 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f41080629a0 con 0x7f40f00ac180 2024-03-26T06:36:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.231+0000 7f4110237700 1 -- 172.21.15.38:0/4262812160 >> [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871] conn(0x7f40f00ac180 msgr2=0x7f40f00ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.232+0000 7f4110237700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871] conn(0x7f40f00ac180 0x7f40f00ae5a0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.232+0000 7f4110237700 1 -- 172.21.15.38:0/4262812160 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f40f00a29a0 msgr2=0x7f40f00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.232+0000 7f4110237700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f40f00a29a0 0x7f40f00a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4108078180 tx=0x7f40fc009190 comp rx=0 tx=0).stop 2024-03-26T06:36:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.232+0000 7f4110237700 1 -- 172.21.15.38:0/4262812160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4108069a90 msgr2=0x7f4108198590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.232+0000 7f4110237700 1 --2- 172.21.15.38:0/4262812160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4108069a90 0x7f4108198590 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f40f80060b0 tx=0x7f40f800d3b0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.232+0000 7f410e7d4700 1 -- 172.21.15.38:0/4262812160 reap_dead start 2024-03-26T06:36:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.232+0000 7f4110237700 1 -- 172.21.15.38:0/4262812160 shutdown_connections 2024-03-26T06:36:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.232+0000 7f4110237700 1 -- 172.21.15.38:0/4262812160 >> 172.21.15.38:0/4262812160 conn(0x7f4108077c10 msgr2=0x7f41080ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:10.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.232+0000 7f4110237700 1 -- 172.21.15.38:0/4262812160 shutdown_connections 2024-03-26T06:36:10.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.232+0000 7f4110237700 1 -- 172.21.15.38:0/4262812160 wait complete. 2024-03-26T06:36:10.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.606+0000 7efdef38d700 1 -- 172.21.15.38:0/1488207243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efde8107e30 msgr2=0x7efde8108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.606+0000 7efdef38d700 1 --2- 172.21.15.38:0/1488207243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efde8107e30 0x7efde8108210 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7efde40099c0 tx=0x7efde4009cd0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.606+0000 7efdef38d700 1 -- 172.21.15.38:0/1488207243 shutdown_connections 2024-03-26T06:36:10.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.606+0000 7efdef38d700 1 --2- 172.21.15.38:0/1488207243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efde806a430 0x7efde810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.606+0000 7efdef38d700 1 --2- 172.21.15.38:0/1488207243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efde8107e30 0x7efde8108210 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.606+0000 7efdef38d700 1 --2- 172.21.15.38:0/1488207243 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efde8069a90 0x7efde8069ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.606+0000 7efdef38d700 1 -- 172.21.15.38:0/1488207243 >> 172.21.15.38:0/1488207243 conn(0x7efde8077c10 msgr2=0x7efde8078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:10.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.606+0000 7efdef38d700 1 -- 172.21.15.38:0/1488207243 shutdown_connections 2024-03-26T06:36:10.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.606+0000 7efdef38d700 1 -- 172.21.15.38:0/1488207243 wait complete. 2024-03-26T06:36:10.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.606+0000 7efdef38d700 1 Processor -- start 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.606+0000 7efdef38d700 1 -- start start 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdef38d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efde8069a90 0x7efde81985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdef38d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efde806a430 0x7efde8198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdef38d700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efde8107e30 0x7efde819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdef38d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde81120f0 con 0x7efde806a430 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdef38d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde8111f70 con 0x7efde8069a90 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdef38d700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde8112270 con 0x7efde8107e30 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdec928700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efde806a430 0x7efde8198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdec928700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efde806a430 0x7efde8198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42534/0 (socket says 172.21.15.38:42534) 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdec928700 1 -- 172.21.15.38:0/2208340254 learned_addr learned my addr 172.21.15.38:0/2208340254 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efded129700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efde8069a90 0x7efde81985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efded92a700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efde8107e30 0x7efde819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:10.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdec928700 1 -- 172.21.15.38:0/2208340254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efde8107e30 msgr2=0x7efde819ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdec928700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7efde8107e30 0x7efde819ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdec928700 1 -- 172.21.15.38:0/2208340254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efde8069a90 msgr2=0x7efde81985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdec928700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efde8069a90 0x7efde81985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdec928700 1 -- 172.21.15.38:0/2208340254 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde819d5d0 con 0x7efde806a430 2024-03-26T06:36:10.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdec928700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efde806a430 0x7efde8198b00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efdd800b530 tx=0x7efdd800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:10.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdde7fc700 1 -- 172.21.15.38:0/2208340254 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8015020 con 0x7efde806a430 2024-03-26T06:36:10.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdde7fc700 1 -- 172.21.15.38:0/2208340254 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdd8011d10 con 0x7efde806a430 2024-03-26T06:36:10.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdde7fc700 1 -- 172.21.15.38:0/2208340254 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd800f2e0 con 0x7efde806a430 2024-03-26T06:36:10.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdef38d700 1 -- 172.21.15.38:0/2208340254 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde819d8c0 con 0x7efde806a430 2024-03-26T06:36:10.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.607+0000 7efdef38d700 1 -- 172.21.15.38:0/2208340254 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde81a5140 con 0x7efde806a430 2024-03-26T06:36:10.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.608+0000 7efdef38d700 1 -- 172.21.15.38:0/2208340254 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efde8062570 con 0x7efde806a430 2024-03-26T06:36:10.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.610+0000 7efdde7fc700 1 -- 172.21.15.38:0/2208340254 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7efdd8018020 con 0x7efde806a430 2024-03-26T06:36:10.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.610+0000 7efdde7fc700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efdd40a29a0 0x7efdd40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:10.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.610+0000 7efded129700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efdd40a29a0 0x7efdd40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:10.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.610+0000 7efdde7fc700 1 -- 172.21.15.38:0/2208340254 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efdd80ce6d0 con 0x7efde806a430 2024-03-26T06:36:10.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.610+0000 7efdde7fc700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853] conn(0x7efdd40ac180 0x7efdd40ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:10.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.610+0000 7efdde7fc700 1 -- 172.21.15.38:0/2208340254 --> [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efdd40aec50 con 0x7efdd40ac180 2024-03-26T06:36:10.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.611+0000 7efdde7fc700 1 -- 172.21.15.38:0/2208340254 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdd80cebb0 con 0x7efde806a430 2024-03-26T06:36:10.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.611+0000 7efded129700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efdd40a29a0 0x7efdd40a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efde4000c00 tx=0x7efde4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:10.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.611+0000 7efded92a700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853] conn(0x7efdd40ac180 0x7efdd40ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:10.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.611+0000 7efded92a700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853] conn(0x7efdd40ac180 0x7efdd40ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:10.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.612+0000 7efdde7fc700 1 -- 172.21.15.38:0/2208340254 <== osd.10 v2:172.21.15.121:6824/1310211853 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efdd40aec50 con 0x7efdd40ac180 2024-03-26T06:36:10.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.628+0000 7efdef38d700 1 -- 172.21.15.38:0/2208340254 --> [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efde8062970 con 0x7efdd40ac180 2024-03-26T06:36:10.631 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efdde7fc700 1 -- 172.21.15.38:0/2208340254 <== osd.10 v2:172.21.15.121:6824/1310211853 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7efde8062970 con 0x7efdd40ac180 2024-03-26T06:36:10.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efdef38d700 1 -- 172.21.15.38:0/2208340254 >> [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853] conn(0x7efdd40ac180 msgr2=0x7efdd40ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efdef38d700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853] conn(0x7efdd40ac180 0x7efdd40ae5a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efdef38d700 1 -- 172.21.15.38:0/2208340254 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efdd40a29a0 msgr2=0x7efdd40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efdef38d700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7efdd40a29a0 0x7efdd40a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efde4000c00 tx=0x7efde4005fb0 comp rx=0 tx=0).stop 2024-03-26T06:36:10.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efdef38d700 1 -- 172.21.15.38:0/2208340254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efde806a430 msgr2=0x7efde8198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:10.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efdef38d700 1 --2- 172.21.15.38:0/2208340254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efde806a430 0x7efde8198b00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7efdd800b530 tx=0x7efdd800b840 comp rx=0 tx=0).stop 2024-03-26T06:36:10.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efded92a700 1 -- 172.21.15.38:0/2208340254 reap_dead start 2024-03-26T06:36:10.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efdef38d700 1 -- 172.21.15.38:0/2208340254 shutdown_connections 2024-03-26T06:36:10.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efdef38d700 1 -- 172.21.15.38:0/2208340254 >> 172.21.15.38:0/2208340254 conn(0x7efde8077c10 msgr2=0x7efde80ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:10.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efdef38d700 1 -- 172.21.15.38:0/2208340254 shutdown_connections 2024-03-26T06:36:10.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:10.629+0000 7efdef38d700 1 -- 172.21.15.38:0/2208340254 wait complete. 2024-03-26T06:36:11.530 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:11 smithi038 ceph-mon[32265]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:11.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.629+0000 7f663d255700 1 -- 172.21.15.38:0/2968222705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381009e0 msgr2=0x7f6638100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:11.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.629+0000 7f663d255700 1 --2- 172.21.15.38:0/2968222705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381009e0 0x7f6638100dc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f662800b0d0 tx=0x7f662800b3e0 comp rx=0 tx=0).stop 2024-03-26T06:36:11.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.629+0000 7f663d255700 1 -- 172.21.15.38:0/2968222705 shutdown_connections 2024-03-26T06:36:11.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.629+0000 7f663d255700 1 --2- 172.21.15.38:0/2968222705 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66381071b0 0x7f663810edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:11.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.629+0000 7f663d255700 1 --2- 172.21.15.38:0/2968222705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381009e0 0x7f6638100dc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:11.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.629+0000 7f663d255700 1 --2- 172.21.15.38:0/2968222705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6638101390 0x7f6638106ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:11.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.629+0000 7f663d255700 1 -- 172.21.15.38:0/2968222705 >> 172.21.15.38:0/2968222705 conn(0x7f6638076060 msgr2=0x7f6638076470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:11.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.629+0000 7f663d255700 1 -- 172.21.15.38:0/2968222705 shutdown_connections 2024-03-26T06:36:11.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.629+0000 7f663d255700 1 -- 172.21.15.38:0/2968222705 wait complete. 2024-03-26T06:36:11.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.630+0000 7f663d255700 1 Processor -- start 2024-03-26T06:36:11.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.630+0000 7f663d255700 1 -- start start 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.630+0000 7f663d255700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381009e0 0x7f6638101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.630+0000 7f663d255700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6638101390 0x7f6638101ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f663d255700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66381071b0 0x7f66381023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f663d255700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66381121d0 con 0x7f66381009e0 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f663d255700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6638112050 con 0x7f6638101390 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f663d255700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6638112350 con 0x7f66381071b0 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f6636d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381009e0 0x7f6638101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f6636d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381009e0 0x7f6638101960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42546/0 (socket says 172.21.15.38:42546) 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f6636d9d700 1 -- 172.21.15.38:0/2199148309 learned_addr learned my addr 172.21.15.38:0/2199148309 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f663759e700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66381071b0 0x7f66381023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f663659c700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6638101390 0x7f6638101ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f6636d9d700 1 -- 172.21.15.38:0/2199148309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66381071b0 msgr2=0x7f66381023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:11.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f6636d9d700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66381071b0 0x7f66381023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:11.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f6636d9d700 1 -- 172.21.15.38:0/2199148309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6638101390 msgr2=0x7f6638101ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:11.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f6636d9d700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6638101390 0x7f6638101ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:11.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f6636d9d700 1 -- 172.21.15.38:0/2199148309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66381a6b90 con 0x7f66381009e0 2024-03-26T06:36:11.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f663759e700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66381071b0 0x7f66381023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:36:11.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f6636d9d700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381009e0 0x7f6638101960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f662800b820 tx=0x7f6628004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:11.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f662ffff700 1 -- 172.21.15.38:0/2199148309 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628021020 con 0x7f66381009e0 2024-03-26T06:36:11.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f663d255700 1 -- 172.21.15.38:0/2199148309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66381a6e20 con 0x7f66381009e0 2024-03-26T06:36:11.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f663d255700 1 -- 172.21.15.38:0/2199148309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66381a7280 con 0x7f66381009e0 2024-03-26T06:36:11.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f662ffff700 1 -- 172.21.15.38:0/2199148309 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6628007d40 con 0x7f66381009e0 2024-03-26T06:36:11.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.631+0000 7f662ffff700 1 -- 172.21.15.38:0/2199148309 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628005370 con 0x7f66381009e0 2024-03-26T06:36:11.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.632+0000 7f663d255700 1 -- 172.21.15.38:0/2199148309 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f66380624f0 con 0x7f66381009e0 2024-03-26T06:36:11.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.633+0000 7f662ffff700 1 -- 172.21.15.38:0/2199148309 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f6628024020 con 0x7f66381009e0 2024-03-26T06:36:11.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.634+0000 7f662ffff700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f66240a29a0 0x7f66240a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:11.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.634+0000 7f663659c700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f66240a29a0 0x7f66240a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:11.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.634+0000 7f662ffff700 1 -- 172.21.15.38:0/2199148309 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f662800f020 con 0x7f66381009e0 2024-03-26T06:36:11.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.634+0000 7f662ffff700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243] conn(0x7f66240ac180 0x7f66240ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:11.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.634+0000 7f662ffff700 1 -- 172.21.15.38:0/2199148309 --> [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66280020b0 con 0x7f66240ac180 2024-03-26T06:36:11.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.634+0000 7f662ffff700 1 -- 172.21.15.38:0/2199148309 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66280d0470 con 0x7f66381009e0 2024-03-26T06:36:11.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.634+0000 7f663759e700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243] conn(0x7f66240ac180 0x7f66240ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:11.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.635+0000 7f663659c700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f66240a29a0 0x7f66240a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6620005fd0 tx=0x7f6620005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:11.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.635+0000 7f663759e700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243] conn(0x7f66240ac180 0x7f66240ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:11.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.635+0000 7f662ffff700 1 -- 172.21.15.38:0/2199148309 <== osd.0 v2:172.21.15.38:6802/1449059243 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f66280020b0 con 0x7f66240ac180 2024-03-26T06:36:11.656 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.654+0000 7f663d255700 1 -- 172.21.15.38:0/2199148309 --> [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f66380628f0 con 0x7f66240ac180 2024-03-26T06:36:11.657 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f662ffff700 1 -- 172.21.15.38:0/2199148309 <== osd.0 v2:172.21.15.38:6802/1449059243 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f66380628f0 con 0x7f66240ac180 2024-03-26T06:36:11.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f663d255700 1 -- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243] conn(0x7f66240ac180 msgr2=0x7f66240ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:11.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f663d255700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243] conn(0x7f66240ac180 0x7f66240ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:11.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f663d255700 1 -- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f66240a29a0 msgr2=0x7f66240a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:11.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f663d255700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f66240a29a0 0x7f66240a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6620005fd0 tx=0x7f6620005f60 comp rx=0 tx=0).stop 2024-03-26T06:36:11.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f663d255700 1 -- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381009e0 msgr2=0x7f6638101960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:11.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f663d255700 1 --2- 172.21.15.38:0/2199148309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66381009e0 0x7f6638101960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f662800b820 tx=0x7f6628004e60 comp rx=0 tx=0).stop 2024-03-26T06:36:11.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f663759e700 1 -- 172.21.15.38:0/2199148309 reap_dead start 2024-03-26T06:36:11.658 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f663d255700 1 -- 172.21.15.38:0/2199148309 shutdown_connections 2024-03-26T06:36:11.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f663d255700 1 -- 172.21.15.38:0/2199148309 >> 172.21.15.38:0/2199148309 conn(0x7f6638076060 msgr2=0x7f66380fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:11.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f663d255700 1 -- 172.21.15.38:0/2199148309 shutdown_connections 2024-03-26T06:36:11.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:11.655+0000 7f663d255700 1 -- 172.21.15.38:0/2199148309 wait complete. 2024-03-26T06:36:11.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:11 smithi121 ceph-mon[36349]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:11.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:11 smithi170 ceph-mon[36912]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:12.401 INFO:teuthology.orchestra.run.smithi038.stdout:34359738445 2024-03-26T06:36:12.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T06:36:12.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.588+0000 7fce0ebdb700 1 -- 172.21.15.38:0/2250023290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce08102820 msgr2=0x7fce0810ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:12.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.588+0000 7fce0ebdb700 1 --2- 172.21.15.38:0/2250023290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce08102820 0x7fce0810ecb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fce0000b0a0 tx=0x7fce0000b3b0 comp rx=0 tx=0).stop 2024-03-26T06:36:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.588+0000 7fce0ebdb700 1 -- 172.21.15.38:0/2250023290 shutdown_connections 2024-03-26T06:36:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.588+0000 7fce0ebdb700 1 --2- 172.21.15.38:0/2250023290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce08102820 0x7fce0810ecb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.588+0000 7fce0ebdb700 1 --2- 172.21.15.38:0/2250023290 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce08107eb0 0x7fce08108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.588+0000 7fce0ebdb700 1 --2- 172.21.15.38:0/2250023290 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce08101e60 0x7fce081022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.588+0000 7fce0ebdb700 1 -- 172.21.15.38:0/2250023290 >> 172.21.15.38:0/2250023290 conn(0x7fce08078f30 msgr2=0x7fce08079340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.588+0000 7fce0ebdb700 1 -- 172.21.15.38:0/2250023290 shutdown_connections 2024-03-26T06:36:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.588+0000 7fce0ebdb700 1 -- 172.21.15.38:0/2250023290 wait complete. 2024-03-26T06:36:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0ebdb700 1 Processor -- start 2024-03-26T06:36:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0ebdb700 1 -- start start 2024-03-26T06:36:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0ebdb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce08101e60 0x7fce080740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:12.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0ebdb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce08102820 0x7fce08074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:12.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0ebdb700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce08107eb0 0x7fce08074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:12.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0ebdb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce081121d0 con 0x7fce08101e60 2024-03-26T06:36:12.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0ebdb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce08112050 con 0x7fce08102820 2024-03-26T06:36:12.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0ebdb700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce08112350 con 0x7fce08107eb0 2024-03-26T06:36:12.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0c977700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce08101e60 0x7fce080740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:12.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0c977700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce08101e60 0x7fce080740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42562/0 (socket says 172.21.15.38:42562) 2024-03-26T06:36:12.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0c977700 1 -- 172.21.15.38:0/1401660121 learned_addr learned my addr 172.21.15.38:0/1401660121 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:12.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce07fff700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce08102820 0x7fce08074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:12.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0c977700 1 -- 172.21.15.38:0/1401660121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce08107eb0 msgr2=0x7fce08074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:12.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0c977700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fce08107eb0 0x7fce08074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:12.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0c977700 1 -- 172.21.15.38:0/1401660121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce08102820 msgr2=0x7fce08074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:12.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0c977700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce08102820 0x7fce08074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:12.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.589+0000 7fce0c977700 1 -- 172.21.15.38:0/1401660121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce081a6c70 con 0x7fce08101e60 2024-03-26T06:36:12.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.590+0000 7fce0c977700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce08101e60 0x7fce080740e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800b530 tx=0x7fcdf800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:12.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.590+0000 7fce05ffb700 1 -- 172.21.15.38:0/1401660121 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8015020 con 0x7fce08101e60 2024-03-26T06:36:12.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.590+0000 7fce05ffb700 1 -- 172.21.15.38:0/1401660121 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcdf8011d10 con 0x7fce08101e60 2024-03-26T06:36:12.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.590+0000 7fce0ebdb700 1 -- 172.21.15.38:0/1401660121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce081a6f60 con 0x7fce08101e60 2024-03-26T06:36:12.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.590+0000 7fce05ffb700 1 -- 172.21.15.38:0/1401660121 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf8010e50 con 0x7fce08101e60 2024-03-26T06:36:12.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.590+0000 7fce0ebdb700 1 -- 172.21.15.38:0/1401660121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce081a7460 con 0x7fce08101e60 2024-03-26T06:36:12.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.591+0000 7fce0ebdb700 1 -- 172.21.15.38:0/1401660121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fce080624f0 con 0x7fce08101e60 2024-03-26T06:36:12.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.592+0000 7fce05ffb700 1 -- 172.21.15.38:0/1401660121 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fcdf8018020 con 0x7fce08101e60 2024-03-26T06:36:12.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.592+0000 7fce05ffb700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcdf00a2bc0 0x7fcdf00a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:12.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.593+0000 7fce07fff700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcdf00a2bc0 0x7fcdf00a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:12.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.593+0000 7fce05ffb700 1 -- 172.21.15.38:0/1401660121 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcdf80ce850 con 0x7fce08101e60 2024-03-26T06:36:12.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.593+0000 7fce05ffb700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176] conn(0x7fcdf00ac3a0 0x7fcdf00ae7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:12.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.593+0000 7fce05ffb700 1 -- 172.21.15.38:0/1401660121 --> [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcdf8010e50 con 0x7fcdf00ac3a0 2024-03-26T06:36:12.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.593+0000 7fce05ffb700 1 -- 172.21.15.38:0/1401660121 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdf80cec40 con 0x7fce08101e60 2024-03-26T06:36:12.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.593+0000 7fce07fff700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcdf00a2bc0 0x7fcdf00a5080 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc006fd0 tx=0x7fcdfc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:12.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.593+0000 7fce0d178700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176] conn(0x7fcdf00ac3a0 0x7fcdf00ae7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:12.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.594+0000 7fce0d178700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176] conn(0x7fcdf00ac3a0 0x7fcdf00ae7c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:12.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.594+0000 7fce05ffb700 1 -- 172.21.15.38:0/1401660121 <== osd.4 v2:172.21.15.121:6808/1471300176 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcdf8010e50 con 0x7fcdf00ac3a0 2024-03-26T06:36:12.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.611+0000 7fce0ebdb700 1 -- 172.21.15.38:0/1401660121 --> [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fce080628f0 con 0x7fcdf00ac3a0 2024-03-26T06:36:12.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.611+0000 7fce05ffb700 1 -- 172.21.15.38:0/1401660121 <== osd.4 v2:172.21.15.121:6808/1471300176 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fce080628f0 con 0x7fcdf00ac3a0 2024-03-26T06:36:12.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.612+0000 7fce0ebdb700 1 -- 172.21.15.38:0/1401660121 >> [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176] conn(0x7fcdf00ac3a0 msgr2=0x7fcdf00ae7c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:12.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.612+0000 7fce0ebdb700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176] conn(0x7fcdf00ac3a0 0x7fcdf00ae7c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:12.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.612+0000 7fce0ebdb700 1 -- 172.21.15.38:0/1401660121 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcdf00a2bc0 msgr2=0x7fcdf00a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:12.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.612+0000 7fce0ebdb700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcdf00a2bc0 0x7fcdf00a5080 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc006fd0 tx=0x7fcdfc008040 comp rx=0 tx=0).stop 2024-03-26T06:36:12.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.612+0000 7fce0ebdb700 1 -- 172.21.15.38:0/1401660121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce08101e60 msgr2=0x7fce080740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:12.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.612+0000 7fce0ebdb700 1 --2- 172.21.15.38:0/1401660121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce08101e60 0x7fce080740e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcdf800b530 tx=0x7fcdf800b8f0 comp rx=0 tx=0).stop 2024-03-26T06:36:12.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.612+0000 7fce0d178700 1 -- 172.21.15.38:0/1401660121 reap_dead start 2024-03-26T06:36:12.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.612+0000 7fce0ebdb700 1 -- 172.21.15.38:0/1401660121 shutdown_connections 2024-03-26T06:36:12.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.612+0000 7fce0ebdb700 1 -- 172.21.15.38:0/1401660121 >> 172.21.15.38:0/1401660121 conn(0x7fce08078f30 msgr2=0x7fce08100a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:12.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.612+0000 7fce0ebdb700 1 -- 172.21.15.38:0/1401660121 shutdown_connections 2024-03-26T06:36:12.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:12.612+0000 7fce0ebdb700 1 -- 172.21.15.38:0/1401660121 wait complete. 2024-03-26T06:36:13.469 INFO:teuthology.orchestra.run.smithi038.stdout:77309411395 2024-03-26T06:36:13.470 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T06:36:13.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:13 smithi038 ceph-mon[32265]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:13.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:13 smithi121 ceph-mon[36349]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:13.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:13 smithi170 ceph-mon[36912]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:14.352 INFO:teuthology.orchestra.run.smithi038.stdout:146028888122 2024-03-26T06:36:14.352 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T06:36:14.433 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:14.433 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:14.468 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:14.472 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:14.478 INFO:teuthology.orchestra.run.smithi038.stdout:313532612634 2024-03-26T06:36:14.478 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-26T06:36:14.574 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:14.585 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:14.587 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:14.587 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:15.682 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:15 smithi038 ceph-mon[32265]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:15.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:15 smithi121 ceph-mon[36349]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:15.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:15 smithi170 ceph-mon[36912]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:17.567 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:17.790 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:17 smithi038 ceph-mon[32265]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:17.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:17 smithi121 ceph-mon[36349]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:17.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:17 smithi170 ceph-mon[36912]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:19.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:19 smithi038 ceph-mon[32265]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:19.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:19 smithi121 ceph-mon[36349]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:19.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:19 smithi170 ceph-mon[36912]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:20.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.325+0000 7fda89a41700 1 -- 172.21.15.38:0/1847391384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda84101390 msgr2=0x7fda84105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:20.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.325+0000 7fda89a41700 1 --2- 172.21.15.38:0/1847391384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda84101390 0x7fda84105140 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fda7800aaa0 tx=0x7fda7800adb0 comp rx=0 tx=0).stop 2024-03-26T06:36:20.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.325+0000 7fda89a41700 1 -- 172.21.15.38:0/1847391384 shutdown_connections 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.325+0000 7fda89a41700 1 --2- 172.21.15.38:0/1847391384 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda84105870 0x7fda8410d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.325+0000 7fda89a41700 1 --2- 172.21.15.38:0/1847391384 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda841009e0 0x7fda84100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.325+0000 7fda89a41700 1 --2- 172.21.15.38:0/1847391384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda84101390 0x7fda84105140 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.325+0000 7fda89a41700 1 -- 172.21.15.38:0/1847391384 >> 172.21.15.38:0/1847391384 conn(0x7fda84076060 msgr2=0x7fda84076470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.325+0000 7fda89a41700 1 -- 172.21.15.38:0/1847391384 shutdown_connections 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.325+0000 7fda89a41700 1 -- 172.21.15.38:0/1847391384 wait complete. 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.325+0000 7fda89a41700 1 Processor -- start 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.325+0000 7fda89a41700 1 -- start start 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda89a41700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda841009e0 0x7fda841986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda89a41700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda84101390 0x7fda84198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda89a41700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda84105870 0x7fda8419cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:20.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda89a41700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda84110030 con 0x7fda841009e0 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda89a41700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8410feb0 con 0x7fda84101390 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda89a41700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda841101b0 con 0x7fda84105870 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda82ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda841009e0 0x7fda841986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda82ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda841009e0 0x7fda841986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37254/0 (socket says 172.21.15.38:37254) 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda82ffd700 1 -- 172.21.15.38:0/455876995 learned_addr learned my addr 172.21.15.38:0/455876995 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda837fe700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda84105870 0x7fda8419cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda827fc700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda84101390 0x7fda84198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda82ffd700 1 -- 172.21.15.38:0/455876995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda84105870 msgr2=0x7fda8419cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda82ffd700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda84105870 0x7fda8419cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda82ffd700 1 -- 172.21.15.38:0/455876995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda84101390 msgr2=0x7fda84198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda82ffd700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda84101390 0x7fda84198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda82ffd700 1 -- 172.21.15.38:0/455876995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8419d6b0 con 0x7fda841009e0 2024-03-26T06:36:20.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda827fc700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda84101390 0x7fda84198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:20.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda82ffd700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda841009e0 0x7fda841986f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fda7400b500 tx=0x7fda7400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:20.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda88a3f700 1 -- 172.21.15.38:0/455876995 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda74015020 con 0x7fda841009e0 2024-03-26T06:36:20.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda89a41700 1 -- 172.21.15.38:0/455876995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8419d9a0 con 0x7fda841009e0 2024-03-26T06:36:20.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda88a3f700 1 -- 172.21.15.38:0/455876995 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda74011d10 con 0x7fda841009e0 2024-03-26T06:36:20.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda89a41700 1 -- 172.21.15.38:0/455876995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda841a52a0 con 0x7fda841009e0 2024-03-26T06:36:20.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.326+0000 7fda88a3f700 1 -- 172.21.15.38:0/455876995 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7400f2e0 con 0x7fda841009e0 2024-03-26T06:36:20.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.327+0000 7fda89a41700 1 -- 172.21.15.38:0/455876995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda84108ab0 con 0x7fda841009e0 2024-03-26T06:36:20.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.329+0000 7fda88a3f700 1 -- 172.21.15.38:0/455876995 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fda74018020 con 0x7fda841009e0 2024-03-26T06:36:20.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.329+0000 7fda88a3f700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda6c0a2920 0x7fda6c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:20.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.329+0000 7fda827fc700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda6c0a2920 0x7fda6c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:20.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.330+0000 7fda88a3f700 1 -- 172.21.15.38:0/455876995 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fda740ce6d0 con 0x7fda841009e0 2024-03-26T06:36:20.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.330+0000 7fda827fc700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda6c0a2920 0x7fda6c0a4de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fda78006010 tx=0x7fda7800c650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:20.336 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.333+0000 7fda88a3f700 1 -- 172.21.15.38:0/455876995 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fda7401f020 con 0x7fda841009e0 2024-03-26T06:36:20.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.539+0000 7fda89a41700 1 -- 172.21.15.38:0/455876995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fda84199a00 con 0x7fda841009e0 2024-03-26T06:36:20.542 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.540+0000 7fda88a3f700 1 -- 172.21.15.38:0/455876995 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fda7409b7d0 con 0x7fda841009e0 2024-03-26T06:36:20.542 INFO:teuthology.orchestra.run.smithi038.stdout:292057776161 2024-03-26T06:36:20.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 -- 172.21.15.38:0/455876995 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda6c0a2920 msgr2=0x7fda6c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:20.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda6c0a2920 0x7fda6c0a4de0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fda78006010 tx=0x7fda7800c650 comp rx=0 tx=0).stop 2024-03-26T06:36:20.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 -- 172.21.15.38:0/455876995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda841009e0 msgr2=0x7fda841986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:20.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda841009e0 0x7fda841986f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fda7400b500 tx=0x7fda7400b8c0 comp rx=0 tx=0).stop 2024-03-26T06:36:20.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 -- 172.21.15.38:0/455876995 shutdown_connections 2024-03-26T06:36:20.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda84105870 0x7fda8419cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:20.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda841009e0 0x7fda841986f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:20.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda6c0a2920 0x7fda6c0a4de0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:20.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 --2- 172.21.15.38:0/455876995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda84101390 0x7fda84198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:20.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 -- 172.21.15.38:0/455876995 >> 172.21.15.38:0/455876995 conn(0x7fda84076060 msgr2=0x7fda840fe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:20.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 -- 172.21.15.38:0/455876995 shutdown_connections 2024-03-26T06:36:20.545 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:20.542+0000 7fda89a41700 1 -- 172.21.15.38:0/455876995 wait complete. 2024-03-26T06:36:21.143 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.141+0000 7f83e4801700 1 -- 172.21.15.38:0/747696681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100e20 msgr2=0x7f83e0101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:21.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.141+0000 7f83e4801700 1 --2- 172.21.15.38:0/747696681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100e20 0x7f83e0101200 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f83d40099c0 tx=0x7f83d4009cd0 comp rx=0 tx=0).stop 2024-03-26T06:36:21.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.141+0000 7f83e4801700 1 -- 172.21.15.38:0/747696681 shutdown_connections 2024-03-26T06:36:21.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.141+0000 7f83e4801700 1 --2- 172.21.15.38:0/747696681 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e0105f00 0x7f83e010dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:21.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.141+0000 7f83e4801700 1 --2- 172.21.15.38:0/747696681 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100e20 0x7f83e0101200 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:21.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.141+0000 7f83e4801700 1 --2- 172.21.15.38:0/747696681 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e01017d0 0x7f83e0105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:21.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.141+0000 7f83e4801700 1 -- 172.21.15.38:0/747696681 >> 172.21.15.38:0/747696681 conn(0x7f83e00fc6b0 msgr2=0x7f83e00fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:21.144 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.141+0000 7f83e4801700 1 -- 172.21.15.38:0/747696681 shutdown_connections 2024-03-26T06:36:21.145 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.141+0000 7f83e4801700 1 -- 172.21.15.38:0/747696681 wait complete. 2024-03-26T06:36:21.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83e4801700 1 Processor -- start 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83e4801700 1 -- start start 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83e4801700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100e20 0x7f83e0199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83e4801700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e01017d0 0x7f83e0199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83e4801700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e0105f00 0x7f83e01a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83e4801700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e0110ef0 con 0x7f83e0100e20 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83e4801700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e0110d70 con 0x7f83e0105f00 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83e4801700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e0111070 con 0x7f83e01017d0 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83ded9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100e20 0x7f83e0199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83ded9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100e20 0x7f83e0199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37264/0 (socket says 172.21.15.38:37264) 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83ded9d700 1 -- 172.21.15.38:0/2185129139 learned_addr learned my addr 172.21.15.38:0/2185129139 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83df59e700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e0105f00 0x7f83e01a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83de59c700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e01017d0 0x7f83e0199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:21.147 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83ded9d700 1 -- 172.21.15.38:0/2185129139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e01017d0 msgr2=0x7f83e0199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:21.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83ded9d700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e01017d0 0x7f83e0199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:21.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.142+0000 7f83ded9d700 1 -- 172.21.15.38:0/2185129139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e0105f00 msgr2=0x7f83e01a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:21.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.143+0000 7f83ded9d700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e0105f00 0x7f83e01a43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:21.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.143+0000 7f83ded9d700 1 -- 172.21.15.38:0/2185129139 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e01a48f0 con 0x7f83e0100e20 2024-03-26T06:36:21.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.143+0000 7f83ded9d700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100e20 0x7f83e0199330 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f83d4006010 tx=0x7f83d4009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:21.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.143+0000 7f83c7fff700 1 -- 172.21.15.38:0/2185129139 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4027070 con 0x7f83e0100e20 2024-03-26T06:36:21.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.143+0000 7f83c7fff700 1 -- 172.21.15.38:0/2185129139 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f83d4004050 con 0x7f83e0100e20 2024-03-26T06:36:21.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.143+0000 7f83e4801700 1 -- 172.21.15.38:0/2185129139 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e01a4ac0 con 0x7f83e0100e20 2024-03-26T06:36:21.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.143+0000 7f83c7fff700 1 -- 172.21.15.38:0/2185129139 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d4022e10 con 0x7f83e0100e20 2024-03-26T06:36:21.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.143+0000 7f83e4801700 1 -- 172.21.15.38:0/2185129139 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e01a4eb0 con 0x7f83e0100e20 2024-03-26T06:36:21.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.144+0000 7f83e4801700 1 -- 172.21.15.38:0/2185129139 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e0068aa0 con 0x7f83e0100e20 2024-03-26T06:36:21.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.146+0000 7f83c7fff700 1 -- 172.21.15.38:0/2185129139 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f83d402a070 con 0x7f83e0100e20 2024-03-26T06:36:21.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.146+0000 7f83c7fff700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f83c80a2920 0x7f83c80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:21.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.146+0000 7f83de59c700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f83c80a2920 0x7f83c80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:21.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.146+0000 7f83c7fff700 1 -- 172.21.15.38:0/2185129139 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f83d40cfa90 con 0x7f83e0100e20 2024-03-26T06:36:21.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.147+0000 7f83de59c700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f83c80a2920 0x7f83c80a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f83d0005950 tx=0x7f83d00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:21.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.150+0000 7f83c7fff700 1 -- 172.21.15.38:0/2185129139 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f83d4013070 con 0x7f83e0100e20 2024-03-26T06:36:21.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.361+0000 7f83e4801700 1 -- 172.21.15.38:0/2185129139 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f83e002d830 con 0x7f83e0100e20 2024-03-26T06:36:21.364 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.362+0000 7f83c7fff700 1 -- 172.21.15.38:0/2185129139 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f83d409cb10 con 0x7f83e0100e20 2024-03-26T06:36:21.364 INFO:teuthology.orchestra.run.smithi038.stdout:201863462961 2024-03-26T06:36:21.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.363+0000 7f83e4801700 1 -- 172.21.15.38:0/2185129139 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f83c80a2920 msgr2=0x7f83c80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:21.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.363+0000 7f83e4801700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f83c80a2920 0x7f83c80a4de0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f83d0005950 tx=0x7f83d00058e0 comp rx=0 tx=0).stop 2024-03-26T06:36:21.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.364+0000 7f83e4801700 1 -- 172.21.15.38:0/2185129139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100e20 msgr2=0x7f83e0199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:21.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.364+0000 7f83e4801700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100e20 0x7f83e0199330 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f83d4006010 tx=0x7f83d4009f80 comp rx=0 tx=0).stop 2024-03-26T06:36:21.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.364+0000 7f83e4801700 1 -- 172.21.15.38:0/2185129139 shutdown_connections 2024-03-26T06:36:21.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.364+0000 7f83e4801700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f83c80a2920 0x7f83c80a4de0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:21.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.364+0000 7f83e4801700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f83e0105f00 0x7f83e01a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:21.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.364+0000 7f83e4801700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83e0100e20 0x7f83e0199330 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:21.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.364+0000 7f83e4801700 1 --2- 172.21.15.38:0/2185129139 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f83e01017d0 0x7f83e0199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:21.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.364+0000 7f83e4801700 1 -- 172.21.15.38:0/2185129139 >> 172.21.15.38:0/2185129139 conn(0x7f83e00fc6b0 msgr2=0x7f83e00fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:21.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.364+0000 7f83e4801700 1 -- 172.21.15.38:0/2185129139 shutdown_connections 2024-03-26T06:36:21.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:21.364+0000 7f83e4801700 1 -- 172.21.15.38:0/2185129139 wait complete. 2024-03-26T06:36:21.768 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:21 smithi038 ceph-mon[32265]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:21.768 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:21 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/455876995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-26T06:36:21.768 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:21 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2185129139' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T06:36:21.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:21 smithi121 ceph-mon[36349]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:21.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:21 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/455876995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-26T06:36:21.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:21 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2185129139' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T06:36:21.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:21 smithi170 ceph-mon[36912]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:21.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:21 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/455876995' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-26T06:36:21.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:21 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2185129139' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T06:36:22.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.473+0000 7f6931911700 1 -- 172.21.15.38:0/1095338951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f692c106d20 msgr2=0x7f692c107100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:22.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.473+0000 7f6931911700 1 --2- 172.21.15.38:0/1095338951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f692c106d20 0x7f692c107100 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f691c009990 tx=0x7f691c009ca0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.473+0000 7f6931911700 1 -- 172.21.15.38:0/1095338951 shutdown_connections 2024-03-26T06:36:22.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.473+0000 7f6931911700 1 --2- 172.21.15.38:0/1095338951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f692c101680 0x7f692c10db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.473+0000 7f6931911700 1 --2- 172.21.15.38:0/1095338951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f692c106d20 0x7f692c107100 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.473+0000 7f6931911700 1 --2- 172.21.15.38:0/1095338951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f692c100ce0 0x7f692c101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.473+0000 7f6931911700 1 -- 172.21.15.38:0/1095338951 >> 172.21.15.38:0/1095338951 conn(0x7f692c077c10 msgr2=0x7f692c078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:22.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.473+0000 7f6931911700 1 -- 172.21.15.38:0/1095338951 shutdown_connections 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.473+0000 7f6931911700 1 -- 172.21.15.38:0/1095338951 wait complete. 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.474+0000 7f6931911700 1 Processor -- start 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.474+0000 7f6931911700 1 -- start start 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f6931911700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f692c100ce0 0x7f692c198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f6931911700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f692c101680 0x7f692c198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f6931911700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f692c106d20 0x7f692c19ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f6931911700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c110f80 con 0x7f692c100ce0 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f6931911700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c110e00 con 0x7f692c106d20 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f6931911700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692c111100 con 0x7f692c101680 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f692affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f692c100ce0 0x7f692c198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f692affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f692c100ce0 0x7f692c198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37278/0 (socket says 172.21.15.38:37278) 2024-03-26T06:36:22.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f692affd700 1 -- 172.21.15.38:0/2587928474 learned_addr learned my addr 172.21.15.38:0/2587928474 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f692a7fc700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f692c101680 0x7f692c198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f692b7fe700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f692c106d20 0x7f692c19ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f692affd700 1 -- 172.21.15.38:0/2587928474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f692c101680 msgr2=0x7f692c198aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f692affd700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f692c101680 0x7f692c198aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f692affd700 1 -- 172.21.15.38:0/2587928474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f692c106d20 msgr2=0x7f692c19ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f692affd700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f692c106d20 0x7f692c19ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f692affd700 1 -- 172.21.15.38:0/2587928474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f692c19d570 con 0x7f692c100ce0 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f692affd700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f692c100ce0 0x7f692c198560 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f691c000c00 tx=0x7f691c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f693090f700 1 -- 172.21.15.38:0/2587928474 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c027070 con 0x7f692c100ce0 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f693090f700 1 -- 172.21.15.38:0/2587928474 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f691c004050 con 0x7f692c100ce0 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f6931911700 1 -- 172.21.15.38:0/2587928474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f692c19d800 con 0x7f692c100ce0 2024-03-26T06:36:22.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f693090f700 1 -- 172.21.15.38:0/2587928474 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691c00f460 con 0x7f692c100ce0 2024-03-26T06:36:22.828 INFO:teuthology.orchestra.run.smithi038.stdout:167503724598 2024-03-26T06:36:22.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.475+0000 7f6931911700 1 -- 172.21.15.38:0/2587928474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f692c1a50e0 con 0x7f692c100ce0 2024-03-26T06:36:22.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.476+0000 7f6931911700 1 -- 172.21.15.38:0/2587928474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f692c1023d0 con 0x7f692c100ce0 2024-03-26T06:36:22.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.478+0000 7f693090f700 1 -- 172.21.15.38:0/2587928474 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f691c02a070 con 0x7f692c100ce0 2024-03-26T06:36:22.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.478+0000 7f693090f700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69140a2920 0x7f69140a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.478+0000 7f692a7fc700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69140a2920 0x7f69140a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.478+0000 7f693090f700 1 -- 172.21.15.38:0/2587928474 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f691c0cf9a0 con 0x7f692c100ce0 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.479+0000 7f692a7fc700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69140a2920 0x7f69140a4de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6920005950 tx=0x7f69200058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.483+0000 7f693090f700 1 -- 172.21.15.38:0/2587928474 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f691c05e260 con 0x7f692c100ce0 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.691+0000 7f6931911700 1 -- 172.21.15.38:0/2587928474 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f692c1a52b0 con 0x7f692c100ce0 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.692+0000 7f693090f700 1 -- 172.21.15.38:0/2587928474 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f691c013070 con 0x7f692c100ce0 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 -- 172.21.15.38:0/2587928474 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69140a2920 msgr2=0x7f69140a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69140a2920 0x7f69140a4de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6920005950 tx=0x7f69200058e0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 -- 172.21.15.38:0/2587928474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f692c100ce0 msgr2=0x7f692c198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f692c100ce0 0x7f692c198560 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f691c000c00 tx=0x7f691c0047b0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 -- 172.21.15.38:0/2587928474 shutdown_connections 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69140a2920 0x7f69140a4de0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f692c106d20 0x7f692c19ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.829 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f692c100ce0 0x7f692c198560 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 --2- 172.21.15.38:0/2587928474 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f692c101680 0x7f692c198aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:22.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 -- 172.21.15.38:0/2587928474 >> 172.21.15.38:0/2587928474 conn(0x7f692c077c10 msgr2=0x7f692c100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:22.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 -- 172.21.15.38:0/2587928474 shutdown_connections 2024-03-26T06:36:22.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:22.695+0000 7f6931911700 1 -- 172.21.15.38:0/2587928474 wait complete. 2024-03-26T06:36:23.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:23 smithi038 ceph-mon[32265]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:23.787 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:23 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2587928474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T06:36:23.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:23 smithi121 ceph-mon[36349]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:23.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:23 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2587928474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T06:36:23.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:23 smithi170 ceph-mon[36912]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:23.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:23 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2587928474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T06:36:24.373 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776158 got 292057776161 for osd.9 2024-03-26T06:36:24.374 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:25.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:25 smithi038 ceph-mon[32265]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:25.876 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462956 got 201863462961 for osd.6 2024-03-26T06:36:25.877 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:25.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:25 smithi121 ceph-mon[36349]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:25.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:25 smithi170 ceph-mon[36912]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:26.249 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:26.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-26T06:36:26.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-26T06:36:26.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-26T06:36:26.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-26T06:36:26.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-26T06:36:26.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-26T06:36:26.742 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]: dispatch 2024-03-26T06:36:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-03-26T06:36:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-26T06:36:26.743 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:26 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:36:26.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-26T06:36:26.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-26T06:36:26.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-26T06:36:26.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-26T06:36:26.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-26T06:36:26.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-26T06:36:26.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]: dispatch 2024-03-26T06:36:26.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-03-26T06:36:26.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-26T06:36:26.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:26 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:36:26.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-03-26T06:36:26.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-03-26T06:36:26.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-26T06:36:26.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]: dispatch 2024-03-26T06:36:26.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-26T06:36:26.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-26T06:36:26.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]: dispatch 2024-03-26T06:36:26.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-03-26T06:36:26.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-03-26T06:36:26.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:26 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:36:26.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.959+0000 7fe07ed48700 1 -- 172.21.15.38:0/2518189112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe078106e50 msgr2=0x7fe078107230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:26.962 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.959+0000 7fe07ed48700 1 --2- 172.21.15.38:0/2518189112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe078106e50 0x7fe078107230 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe06c009990 tx=0x7fe06c009ca0 comp rx=0 tx=0).stop 2024-03-26T06:36:26.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.960+0000 7fe07ed48700 1 -- 172.21.15.38:0/2518189112 shutdown_connections 2024-03-26T06:36:26.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.960+0000 7fe07ed48700 1 --2- 172.21.15.38:0/2518189112 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe078100d90 0x7fe07810ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:26.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.960+0000 7fe07ed48700 1 --2- 172.21.15.38:0/2518189112 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe078106e50 0x7fe078107230 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:26.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.960+0000 7fe07ed48700 1 --2- 172.21.15.38:0/2518189112 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0781003d0 0x7fe078100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:26.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.960+0000 7fe07ed48700 1 -- 172.21.15.38:0/2518189112 >> 172.21.15.38:0/2518189112 conn(0x7fe0780772e0 msgr2=0x7fe0780fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:26.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.960+0000 7fe07ed48700 1 -- 172.21.15.38:0/2518189112 shutdown_connections 2024-03-26T06:36:26.963 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.960+0000 7fe07ed48700 1 -- 172.21.15.38:0/2518189112 wait complete. 2024-03-26T06:36:26.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.960+0000 7fe07ed48700 1 Processor -- start 2024-03-26T06:36:26.964 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.960+0000 7fe07ed48700 1 -- start start 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07ed48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0781003d0 0x7fe0781986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07ed48700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe078100d90 0x7fe078198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07ed48700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe078106e50 0x7fe07819cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07ed48700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe078112230 con 0x7fe078106e50 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07ed48700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0781120b0 con 0x7fe0781003d0 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07ed48700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0781123b0 con 0x7fe078100d90 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07cae4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0781003d0 0x7fe0781986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07cae4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0781003d0 0x7fe0781986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:48720/0 (socket says 172.21.15.38:48720) 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07cae4700 1 -- 172.21.15.38:0/2167081734 learned_addr learned my addr 172.21.15.38:0/2167081734 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07d2e5700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe078106e50 0x7fe07819cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe077fff700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe078100d90 0x7fe078198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07cae4700 1 -- 172.21.15.38:0/2167081734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe078100d90 msgr2=0x7fe078198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07cae4700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe078100d90 0x7fe078198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:26.965 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07cae4700 1 -- 172.21.15.38:0/2167081734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe078106e50 msgr2=0x7fe07819cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:26.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07cae4700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe078106e50 0x7fe07819cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:26.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07cae4700 1 -- 172.21.15.38:0/2167081734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07819d6b0 con 0x7fe0781003d0 2024-03-26T06:36:26.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.961+0000 7fe07d2e5700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe078106e50 0x7fe07819cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:36:26.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.962+0000 7fe07cae4700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0781003d0 0x7fe0781986a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe06c000c00 tx=0x7fe06c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:26.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.962+0000 7fe075ffb700 1 -- 172.21.15.38:0/2167081734 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c027070 con 0x7fe0781003d0 2024-03-26T06:36:26.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.962+0000 7fe07ed48700 1 -- 172.21.15.38:0/2167081734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07819d940 con 0x7fe0781003d0 2024-03-26T06:36:26.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.962+0000 7fe075ffb700 1 -- 172.21.15.38:0/2167081734 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe06c003d70 con 0x7fe0781003d0 2024-03-26T06:36:26.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.962+0000 7fe075ffb700 1 -- 172.21.15.38:0/2167081734 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06c00f460 con 0x7fe0781003d0 2024-03-26T06:36:26.966 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.962+0000 7fe07ed48700 1 -- 172.21.15.38:0/2167081734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0781a5220 con 0x7fe0781003d0 2024-03-26T06:36:26.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.963+0000 7fe07ed48700 1 -- 172.21.15.38:0/2167081734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07810a420 con 0x7fe0781003d0 2024-03-26T06:36:26.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.964+0000 7fe075ffb700 1 -- 172.21.15.38:0/2167081734 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe06c02a070 con 0x7fe0781003d0 2024-03-26T06:36:26.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.965+0000 7fe075ffb700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0680a2920 0x7fe0680a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:26.967 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.965+0000 7fe077fff700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0680a2920 0x7fe0680a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:26.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.965+0000 7fe075ffb700 1 -- 172.21.15.38:0/2167081734 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe06c0cfa70 con 0x7fe0781003d0 2024-03-26T06:36:26.968 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.965+0000 7fe077fff700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0680a2920 0x7fe0680a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe0780ff8d0 tx=0x7fe064009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:26.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:26.968+0000 7fe075ffb700 1 -- 172.21.15.38:0/2167081734 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe06c013070 con 0x7fe0781003d0 2024-03-26T06:36:27.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.157+0000 7f74c608b700 1 -- 172.21.15.38:0/2996182159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c0100ce0 msgr2=0x7f74c0101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:27.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.157+0000 7f74c608b700 1 --2- 172.21.15.38:0/2996182159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c0100ce0 0x7f74c0101140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009990 tx=0x7f74b4009ca0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.157+0000 7f74c608b700 1 -- 172.21.15.38:0/2996182159 shutdown_connections 2024-03-26T06:36:27.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.157+0000 7f74c608b700 1 --2- 172.21.15.38:0/2996182159 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74c0101680 0x7f74c010db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.157+0000 7f74c608b700 1 --2- 172.21.15.38:0/2996182159 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74c0106d20 0x7f74c0107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.157+0000 7f74c608b700 1 --2- 172.21.15.38:0/2996182159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c0100ce0 0x7f74c0101140 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.157+0000 7f74c608b700 1 -- 172.21.15.38:0/2996182159 >> 172.21.15.38:0/2996182159 conn(0x7f74c0077c10 msgr2=0x7f74c0078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:27.180 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.157+0000 7f74c608b700 1 -- 172.21.15.38:0/2996182159 shutdown_connections 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.157+0000 7f74c608b700 1 -- 172.21.15.38:0/2996182159 wait complete. 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.158+0000 7f74c608b700 1 Processor -- start 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.158+0000 7f74c608b700 1 -- start start 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.158+0000 7f74c608b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c0100ce0 0x7f74c0199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.158+0000 7f74c608b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74c0101680 0x7f74c0199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.158+0000 7f74c608b700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74c0106d20 0x7f74c01a44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.158+0000 7f74c608b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c0111010 con 0x7f74c0100ce0 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74c608b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c0110e90 con 0x7f74c0101680 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74c608b700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c0111190 con 0x7f74c0106d20 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c0100ce0 0x7f74c0199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bf7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c0100ce0 0x7f74c0199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37304/0 (socket says 172.21.15.38:37304) 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bf7fe700 1 -- 172.21.15.38:0/3401150215 learned_addr learned my addr 172.21.15.38:0/3401150215 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:27.181 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bffff700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74c0106d20 0x7f74c01a44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74beffd700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74c0101680 0x7f74c0199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bf7fe700 1 -- 172.21.15.38:0/3401150215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74c0106d20 msgr2=0x7f74c01a44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bf7fe700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74c0106d20 0x7f74c01a44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bf7fe700 1 -- 172.21.15.38:0/3401150215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74c0101680 msgr2=0x7f74c0199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bf7fe700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74c0101680 0x7f74c0199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bf7fe700 1 -- 172.21.15.38:0/3401150215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c01a4a10 con 0x7f74c0100ce0 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bffff700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74c0106d20 0x7f74c01a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bf7fe700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c0100ce0 0x7f74c0199450 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f74b000e9a0 tx=0x7f74b000ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bcff9700 1 -- 172.21.15.38:0/3401150215 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b000cbd0 con 0x7f74c0100ce0 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74c608b700 1 -- 172.21.15.38:0/3401150215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c01a4be0 con 0x7f74c0100ce0 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bcff9700 1 -- 172.21.15.38:0/3401150215 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74b000cd70 con 0x7f74c0100ce0 2024-03-26T06:36:27.182 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.159+0000 7f74bcff9700 1 -- 172.21.15.38:0/3401150215 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b00187c0 con 0x7f74c0100ce0 2024-03-26T06:36:27.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.160+0000 7f74c608b700 1 -- 172.21.15.38:0/3401150215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c01a50f0 con 0x7f74c0100ce0 2024-03-26T06:36:27.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.161+0000 7f74c608b700 1 -- 172.21.15.38:0/3401150215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c0068aa0 con 0x7f74c0100ce0 2024-03-26T06:36:27.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.162+0000 7f74bcff9700 1 -- 172.21.15.38:0/3401150215 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f74b0010430 con 0x7f74c0100ce0 2024-03-26T06:36:27.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.163+0000 7f74bcff9700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74a80a2920 0x7f74a80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:27.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.163+0000 7f74beffd700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74a80a2920 0x7f74a80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:27.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.163+0000 7f74bcff9700 1 -- 172.21.15.38:0/3401150215 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f74b00cf000 con 0x7f74c0100ce0 2024-03-26T06:36:27.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.163+0000 7f74beffd700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74a80a2920 0x7f74a80a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f74b400b2e0 tx=0x7f74b401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:27.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.172+0000 7f74bcff9700 1 -- 172.21.15.38:0/3401150215 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f74b0053260 con 0x7f74c0100ce0 2024-03-26T06:36:27.186 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.184+0000 7fe07ed48700 1 -- 172.21.15.38:0/2167081734 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fe078068aa0 con 0x7fe0781003d0 2024-03-26T06:36:27.187 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.185+0000 7fe075ffb700 1 -- 172.21.15.38:0/2167081734 <== mon.2 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe06c09ca00 con 0x7fe0781003d0 2024-03-26T06:36:27.187 INFO:teuthology.orchestra.run.smithi038.stdout:223338299437 2024-03-26T06:36:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 -- 172.21.15.38:0/2167081734 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0680a2920 msgr2=0x7fe0680a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0680a2920 0x7fe0680a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe0780ff8d0 tx=0x7fe064009260 comp rx=0 tx=0).stop 2024-03-26T06:36:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 -- 172.21.15.38:0/2167081734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0781003d0 msgr2=0x7fe0781986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0781003d0 0x7fe0781986a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe06c000c00 tx=0x7fe06c00bab0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 -- 172.21.15.38:0/2167081734 shutdown_connections 2024-03-26T06:36:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe078106e50 0x7fe07819cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0680a2920 0x7fe0680a4de0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0781003d0 0x7fe0781986a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.191 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 --2- 172.21.15.38:0/2167081734 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe078100d90 0x7fe078198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 -- 172.21.15.38:0/2167081734 >> 172.21.15.38:0/2167081734 conn(0x7fe0780772e0 msgr2=0x7fe078104ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 -- 172.21.15.38:0/2167081734 shutdown_connections 2024-03-26T06:36:27.192 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.188+0000 7fe07ed48700 1 -- 172.21.15.38:0/2167081734 wait complete. 2024-03-26T06:36:27.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.385+0000 7f74c608b700 1 -- 172.21.15.38:0/3401150215 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f74c002d9f0 con 0x7f74c0100ce0 2024-03-26T06:36:27.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.385+0000 7f74bcff9700 1 -- 172.21.15.38:0/3401150215 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f74b009bf40 con 0x7f74c0100ce0 2024-03-26T06:36:27.388 INFO:teuthology.orchestra.run.smithi038.stdout:77309411399 2024-03-26T06:36:27.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.388+0000 7f74c608b700 1 -- 172.21.15.38:0/3401150215 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74a80a2920 msgr2=0x7f74a80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:27.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.388+0000 7f74c608b700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74a80a2920 0x7f74a80a4de0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f74b400b2e0 tx=0x7f74b401a040 comp rx=0 tx=0).stop 2024-03-26T06:36:27.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.388+0000 7f74c608b700 1 -- 172.21.15.38:0/3401150215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c0100ce0 msgr2=0x7f74c0199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:27.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.388+0000 7f74c608b700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c0100ce0 0x7f74c0199450 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f74b000e9a0 tx=0x7f74b000ed60 comp rx=0 tx=0).stop 2024-03-26T06:36:27.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.389+0000 7f74c608b700 1 -- 172.21.15.38:0/3401150215 shutdown_connections 2024-03-26T06:36:27.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.389+0000 7f74c608b700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f74c0106d20 0x7f74c01a44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.389+0000 7f74c608b700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74c0100ce0 0x7f74c0199450 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.389+0000 7f74c608b700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f74a80a2920 0x7f74a80a4de0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.389+0000 7f74c608b700 1 --2- 172.21.15.38:0/3401150215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f74c0101680 0x7f74c0199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:27.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.389+0000 7f74c608b700 1 -- 172.21.15.38:0/3401150215 >> 172.21.15.38:0/3401150215 conn(0x7f74c0077c10 msgr2=0x7f74c00ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:27.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.389+0000 7f74c608b700 1 -- 172.21.15.38:0/3401150215 shutdown_connections 2024-03-26T06:36:27.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:27.389+0000 7f74c608b700 1 -- 172.21.15.38:0/3401150215 wait complete. 2024-03-26T06:36:27.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-26T06:36:27.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-26T06:36:27.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-26T06:36:27.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-26T06:36:27.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-26T06:36:27.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-26T06:36:27.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-26T06:36:27.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]': finished 2024-03-26T06:36:27.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-03-26T06:36:27.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-26T06:36:27.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: osdmap e87: 12 total, 12 up, 12 in 2024-03-26T06:36:27.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2167081734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T06:36:27.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:27 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3401150215' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T06:36:27.739 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-26T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-26T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-26T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-26T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-26T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-26T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-26T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]': finished 2024-03-26T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-03-26T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-26T06:36:27.740 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: osdmap e87: 12 total, 12 up, 12 in 2024-03-26T06:36:27.741 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2167081734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T06:36:27.741 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:27 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3401150215' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T06:36:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-26T06:36:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-03-26T06:36:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-03-26T06:36:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-26T06:36:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 11]}]': finished 2024-03-26T06:36:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-26T06:36:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-26T06:36:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2d", "id": [3, 9]}]': finished 2024-03-26T06:36:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-03-26T06:36:27.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-03-26T06:36:27.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: osdmap e87: 12 total, 12 up, 12 in 2024-03-26T06:36:27.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2167081734' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T06:36:27.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:27 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3401150215' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T06:36:27.960 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724594 got 167503724598 for osd.5 2024-03-26T06:36:27.961 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:28.773 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411395 got 77309411399 for osd.2 2024-03-26T06:36:28.774 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:28.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:28 smithi038 ceph-mon[32265]: osdmap e88: 12 total, 12 up, 12 in 2024-03-26T06:36:28.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:28 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:28.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:28 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:28.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:28 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:28.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:28 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:28.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:28 smithi121 ceph-mon[36349]: osdmap e88: 12 total, 12 up, 12 in 2024-03-26T06:36:28.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:28 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:28.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:28 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:28.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:28 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:28.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:28 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:28.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:28 smithi170 ceph-mon[36912]: osdmap e88: 12 total, 12 up, 12 in 2024-03-26T06:36:28.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:28 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:28.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:28 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:28.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:28 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:28.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:28 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:30.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:29.998+0000 7f1c7a721700 1 -- 172.21.15.38:0/1614908447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c74100cc0 msgr2=0x7f1c74101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:29.998+0000 7f1c7a721700 1 --2- 172.21.15.38:0/1614908447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c74100cc0 0x7f1c74101140 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1c68009990 tx=0x7f1c68009ca0 comp rx=0 tx=0).stop 2024-03-26T06:36:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:29.998+0000 7f1c7a721700 1 -- 172.21.15.38:0/1614908447 shutdown_connections 2024-03-26T06:36:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:29.998+0000 7f1c7a721700 1 --2- 172.21.15.38:0/1614908447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c74101680 0x7f1c7410db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:29.998+0000 7f1c7a721700 1 --2- 172.21.15.38:0/1614908447 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c74106d70 0x7f1c74107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:29.998+0000 7f1c7a721700 1 --2- 172.21.15.38:0/1614908447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c74100cc0 0x7f1c74101140 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:29.998+0000 7f1c7a721700 1 -- 172.21.15.38:0/1614908447 >> 172.21.15.38:0/1614908447 conn(0x7f1c74077cf0 msgr2=0x7f1c74078100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:29.998+0000 7f1c7a721700 1 -- 172.21.15.38:0/1614908447 shutdown_connections 2024-03-26T06:36:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:29.998+0000 7f1c7a721700 1 -- 172.21.15.38:0/1614908447 wait complete. 2024-03-26T06:36:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:29.999+0000 7f1c7a721700 1 Processor -- start 2024-03-26T06:36:30.002 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:29.999+0000 7f1c7a721700 1 -- start start 2024-03-26T06:36:30.004 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c7a721700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c74100cc0 0x7f1c74197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c7a721700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c74101680 0x7f1c74197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c7a721700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c74106d70 0x7f1c7419be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c7a721700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c74111060 con 0x7f1c74106d70 2024-03-26T06:36:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c7a721700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c74110ee0 con 0x7f1c74100cc0 2024-03-26T06:36:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c7a721700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c741111e0 con 0x7f1c74101680 2024-03-26T06:36:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c78cbe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c74106d70 0x7f1c7419be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c78cbe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c74106d70 0x7f1c7419be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37326/0 (socket says 172.21.15.38:37326) 2024-03-26T06:36:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c78cbe700 1 -- 172.21.15.38:0/3935137585 learned_addr learned my addr 172.21.15.38:0/3935137585 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c737fe700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c74101680 0x7f1c74197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:30.005 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c73fff700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c74100cc0 0x7f1c74197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c78cbe700 1 -- 172.21.15.38:0/3935137585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c74101680 msgr2=0x7f1c74197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c78cbe700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c74101680 0x7f1c74197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c78cbe700 1 -- 172.21.15.38:0/3935137585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c74100cc0 msgr2=0x7f1c74197530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c78cbe700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c74100cc0 0x7f1c74197530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c78cbe700 1 -- 172.21.15.38:0/3935137585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7419c540 con 0x7f1c74106d70 2024-03-26T06:36:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c73fff700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c74100cc0 0x7f1c74197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:36:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c78cbe700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c74106d70 0x7f1c7419be00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c007ee0 tx=0x7f1c6c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c717fa700 1 -- 172.21.15.38:0/3935137585 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c00dbd0 con 0x7f1c74106d70 2024-03-26T06:36:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c717fa700 1 -- 172.21.15.38:0/3935137585 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c6c00dd70 con 0x7f1c74106d70 2024-03-26T06:36:30.006 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.000+0000 7f1c7a721700 1 -- 172.21.15.38:0/3935137585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7419c830 con 0x7f1c74106d70 2024-03-26T06:36:30.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.001+0000 7f1c717fa700 1 -- 172.21.15.38:0/3935137585 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6c005a60 con 0x7f1c74106d70 2024-03-26T06:36:30.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.001+0000 7f1c7a721700 1 -- 172.21.15.38:0/3935137585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c741a40b0 con 0x7f1c74106d70 2024-03-26T06:36:30.007 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.001+0000 7f1c7a721700 1 -- 172.21.15.38:0/3935137585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c74068aa0 con 0x7f1c74106d70 2024-03-26T06:36:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.003+0000 7f1c717fa700 1 -- 172.21.15.38:0/3935137585 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f1c6c01a020 con 0x7f1c74106d70 2024-03-26T06:36:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.003+0000 7f1c717fa700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c5c0a29a0 0x7f1c5c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.003+0000 7f1c73fff700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c5c0a29a0 0x7f1c5c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.003+0000 7f1c717fa700 1 -- 172.21.15.38:0/3935137585 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1c6c0ce2a0 con 0x7f1c74106d70 2024-03-26T06:36:30.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.004+0000 7f1c73fff700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c5c0a29a0 0x7f1c5c0a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1c640099d0 tx=0x7f1c64008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:30.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.008+0000 7f1c717fa700 1 -- 172.21.15.38:0/3935137585 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1c6c010070 con 0x7f1c74106d70 2024-03-26T06:36:30.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.219+0000 7f1c7a721700 1 -- 172.21.15.38:0/3935137585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1c7402d830 con 0x7f1c74106d70 2024-03-26T06:36:30.222 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.219+0000 7f1c717fa700 1 -- 172.21.15.38:0/3935137585 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1c6c09b1e0 con 0x7f1c74106d70 2024-03-26T06:36:30.222 INFO:teuthology.orchestra.run.smithi038.stdout:55834574924 2024-03-26T06:36:30.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 -- 172.21.15.38:0/3935137585 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c5c0a29a0 msgr2=0x7f1c5c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:30.224 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c5c0a29a0 0x7f1c5c0a4e60 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1c640099d0 tx=0x7f1c64008040 comp rx=0 tx=0).stop 2024-03-26T06:36:30.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 -- 172.21.15.38:0/3935137585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c74106d70 msgr2=0x7f1c7419be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:30.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c74106d70 0x7f1c7419be00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c007ee0 tx=0x7f1c6c00d3b0 comp rx=0 tx=0).stop 2024-03-26T06:36:30.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 -- 172.21.15.38:0/3935137585 shutdown_connections 2024-03-26T06:36:30.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c74106d70 0x7f1c7419be00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:30.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c5c0a29a0 0x7f1c5c0a4e60 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:30.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c74100cc0 0x7f1c74197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:30.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 --2- 172.21.15.38:0/3935137585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c74101680 0x7f1c74197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:30.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 -- 172.21.15.38:0/3935137585 >> 172.21.15.38:0/3935137585 conn(0x7f1c74077cf0 msgr2=0x7f1c740ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:30.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 -- 172.21.15.38:0/3935137585 shutdown_connections 2024-03-26T06:36:30.225 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:30.222+0000 7f1c7a721700 1 -- 172.21.15.38:0/3935137585 wait complete. 2024-03-26T06:36:30.465 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:30 smithi038 ceph-mon[32265]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-26T06:36:30.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:30 smithi121 ceph-mon[36349]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-26T06:36:30.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:30 smithi170 ceph-mon[36912]: pgmap v319: 97 pgs: 1 peering, 96 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-26T06:36:30.890 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299432 got 223338299437 for osd.7 2024-03-26T06:36:31.037 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:31.547 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:31 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3935137585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T06:36:31.581 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574918 got 55834574924 for osd.1 2024-03-26T06:36:31.581 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:31.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:31 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3935137585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T06:36:31.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:31 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3935137585' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T06:36:32.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:32 smithi038 ceph-mon[32265]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T06:36:32.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:32 smithi038 ceph-mon[32265]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T06:36:32.585 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.579+0000 7fd3295a1700 1 -- 172.21.15.38:0/306237496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd324112cd0 msgr2=0x7fd3241150c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:32.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.579+0000 7fd3295a1700 1 --2- 172.21.15.38:0/306237496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd324112cd0 0x7fd3241150c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd31c00b0a0 tx=0x7fd31c00b3b0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.580+0000 7fd3295a1700 1 -- 172.21.15.38:0/306237496 shutdown_connections 2024-03-26T06:36:32.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.580+0000 7fd3295a1700 1 --2- 172.21.15.38:0/306237496 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd324112cd0 0x7fd3241150c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.580+0000 7fd3295a1700 1 --2- 172.21.15.38:0/306237496 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd324074180 0x7fd324074560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.580+0000 7fd3295a1700 1 --2- 172.21.15.38:0/306237496 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd324074aa0 0x7fd324112790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.586 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.580+0000 7fd3295a1700 1 -- 172.21.15.38:0/306237496 >> 172.21.15.38:0/306237496 conn(0x7fd3240fd7d0 msgr2=0x7fd3240ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:32.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.580+0000 7fd3295a1700 1 -- 172.21.15.38:0/306237496 shutdown_connections 2024-03-26T06:36:32.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.580+0000 7fd3295a1700 1 -- 172.21.15.38:0/306237496 wait complete. 2024-03-26T06:36:32.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.580+0000 7fd3295a1700 1 Processor -- start 2024-03-26T06:36:32.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.580+0000 7fd3295a1700 1 -- start start 2024-03-26T06:36:32.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3295a1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd324074180 0x7fd3241a46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:32.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3295a1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd324074aa0 0x7fd3241a4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:32.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3295a1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd324112cd0 0x7fd32419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:32.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3295a1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd324118640 con 0x7fd324074aa0 2024-03-26T06:36:32.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3295a1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3241184c0 con 0x7fd324074180 2024-03-26T06:36:32.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3295a1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3241187c0 con 0x7fd324112cd0 2024-03-26T06:36:32.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3237fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd324074aa0 0x7fd3241a4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:32.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3237fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd324074aa0 0x7fd3241a4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37346/0 (socket says 172.21.15.38:37346) 2024-03-26T06:36:32.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3237fe700 1 -- 172.21.15.38:0/22357481 learned_addr learned my addr 172.21.15.38:0/22357481 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:32.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd323fff700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd324074180 0x7fd3241a46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:32.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd328da0700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd324112cd0 0x7fd32419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:32.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3237fe700 1 -- 172.21.15.38:0/22357481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd324112cd0 msgr2=0x7fd32419e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:32.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3237fe700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd324112cd0 0x7fd32419e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3237fe700 1 -- 172.21.15.38:0/22357481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd324074180 msgr2=0x7fd3241a46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:32.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3237fe700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd324074180 0x7fd3241a46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3237fe700 1 -- 172.21.15.38:0/22357481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32419f020 con 0x7fd324074aa0 2024-03-26T06:36:32.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3237fe700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd324074aa0 0x7fd3241a4c00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd31800c9b0 tx=0x7fd31800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:32.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.581+0000 7fd3217fa700 1 -- 172.21.15.38:0/22357481 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318007b00 con 0x7fd324074aa0 2024-03-26T06:36:32.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.582+0000 7fd3217fa700 1 -- 172.21.15.38:0/22357481 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd318007ca0 con 0x7fd324074aa0 2024-03-26T06:36:32.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.582+0000 7fd3295a1700 1 -- 172.21.15.38:0/22357481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32419f310 con 0x7fd324074aa0 2024-03-26T06:36:32.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.582+0000 7fd3217fa700 1 -- 172.21.15.38:0/22357481 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318018730 con 0x7fd324074aa0 2024-03-26T06:36:32.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.582+0000 7fd3295a1700 1 -- 172.21.15.38:0/22357481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3241ab520 con 0x7fd324074aa0 2024-03-26T06:36:32.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.583+0000 7fd3295a1700 1 -- 172.21.15.38:0/22357481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3240758c0 con 0x7fd324074aa0 2024-03-26T06:36:32.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.584+0000 7fd3217fa700 1 -- 172.21.15.38:0/22357481 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fd3180188d0 con 0x7fd324074aa0 2024-03-26T06:36:32.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.584+0000 7fd3217fa700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd30c0a29a0 0x7fd30c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:32.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.584+0000 7fd323fff700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd30c0a29a0 0x7fd30c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:32.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.584+0000 7fd3217fa700 1 -- 172.21.15.38:0/22357481 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd3180ceb40 con 0x7fd324074aa0 2024-03-26T06:36:32.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.585+0000 7fd323fff700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd30c0a29a0 0x7fd30c0a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd314009a10 tx=0x7fd314009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:32.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.589+0000 7fd3217fa700 1 -- 172.21.15.38:0/22357481 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd318014030 con 0x7fd324074aa0 2024-03-26T06:36:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:32 smithi121 ceph-mon[36349]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T06:36:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:32 smithi121 ceph-mon[36349]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T06:36:32.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:32 smithi170 ceph-mon[36912]: pgmap v320: 97 pgs: 1 peering, 96 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T06:36:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:32 smithi170 ceph-mon[36912]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T06:36:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.782+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/622964701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ea4100ce0 msgr2=0x7f4ea4101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.782+0000 7f4eaaaa1700 1 --2- 172.21.15.38:0/622964701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ea4100ce0 0x7f4ea4101140 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009990 tx=0x7f4e90009ca0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.782+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/622964701 shutdown_connections 2024-03-26T06:36:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.782+0000 7f4eaaaa1700 1 --2- 172.21.15.38:0/622964701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ea4101680 0x7f4ea410db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.782+0000 7f4eaaaa1700 1 --2- 172.21.15.38:0/622964701 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ea4106d20 0x7f4ea4107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.782+0000 7f4eaaaa1700 1 --2- 172.21.15.38:0/622964701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ea4100ce0 0x7f4ea4101140 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.782+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/622964701 >> 172.21.15.38:0/622964701 conn(0x7f4ea4077c10 msgr2=0x7f4ea4078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.782+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/622964701 shutdown_connections 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.782+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/622964701 wait complete. 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4eaaaa1700 1 Processor -- start 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4eaaaa1700 1 -- start start 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4eaaaa1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ea4100ce0 0x7f4ea4198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4eaaaa1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ea4101680 0x7f4ea4198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4eaaaa1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ea4106d20 0x7f4ea419cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4eaaaa1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4110f80 con 0x7f4ea4101680 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4eaaaa1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4110e00 con 0x7f4ea4106d20 2024-03-26T06:36:32.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4eaaaa1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea4111100 con 0x7f4ea4100ce0 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ea4101680 0x7f4ea4198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ea4101680 0x7f4ea4198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37354/0 (socket says 172.21.15.38:37354) 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea3fff700 1 -- 172.21.15.38:0/700080285 learned_addr learned my addr 172.21.15.38:0/700080285 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea903e700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ea4106d20 0x7f4ea419cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea883d700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ea4100ce0 0x7f4ea4198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea3fff700 1 -- 172.21.15.38:0/700080285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ea4100ce0 msgr2=0x7f4ea4198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea3fff700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ea4100ce0 0x7f4ea4198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea3fff700 1 -- 172.21.15.38:0/700080285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ea4106d20 msgr2=0x7f4ea419cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea3fff700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ea4106d20 0x7f4ea419cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea3fff700 1 -- 172.21.15.38:0/700080285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea419d510 con 0x7f4ea4101680 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea903e700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ea4106d20 0x7f4ea419cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:32.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.783+0000 7f4ea3fff700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ea4101680 0x7f4ea4198a40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4e90000c00 tx=0x7f4e9000bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.784+0000 7f4ea1ffb700 1 -- 172.21.15.38:0/700080285 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e90027070 con 0x7f4ea4101680 2024-03-26T06:36:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.784+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/700080285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea419d7a0 con 0x7f4ea4101680 2024-03-26T06:36:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.784+0000 7f4ea1ffb700 1 -- 172.21.15.38:0/700080285 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e90022470 con 0x7f4ea4101680 2024-03-26T06:36:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.784+0000 7f4ea1ffb700 1 -- 172.21.15.38:0/700080285 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9000f6a0 con 0x7f4ea4101680 2024-03-26T06:36:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.784+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/700080285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea41a5080 con 0x7f4ea4101680 2024-03-26T06:36:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.785+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/700080285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ea4068aa0 con 0x7f4ea4101680 2024-03-26T06:36:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.786+0000 7f4ea1ffb700 1 -- 172.21.15.38:0/700080285 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f4e9002a070 con 0x7f4ea4101680 2024-03-26T06:36:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.787+0000 7f4ea1ffb700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4e940a2920 0x7f4e940a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.787+0000 7f4ea883d700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4e940a2920 0x7f4e940a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.787+0000 7f4ea1ffb700 1 -- 172.21.15.38:0/700080285 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4e900cff50 con 0x7f4ea4101680 2024-03-26T06:36:32.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.787+0000 7f4ea883d700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4e940a2920 0x7f4e940a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009b10 tx=0x7f4e98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:32.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.791+0000 7f4ea1ffb700 1 -- 172.21.15.38:0/700080285 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4e90013070 con 0x7f4ea4101680 2024-03-26T06:36:32.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.798+0000 7fd3295a1700 1 -- 172.21.15.38:0/22357481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fd32402d830 con 0x7fd324074aa0 2024-03-26T06:36:32.801 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.799+0000 7fd3217fa700 1 -- 172.21.15.38:0/22357481 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fd31809ba80 con 0x7fd324074aa0 2024-03-26T06:36:32.801 INFO:teuthology.orchestra.run.smithi038.stdout:347892351002 2024-03-26T06:36:32.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 -- 172.21.15.38:0/22357481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd30c0a29a0 msgr2=0x7fd30c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:32.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd30c0a29a0 0x7fd30c0a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd314009a10 tx=0x7fd314009190 comp rx=0 tx=0).stop 2024-03-26T06:36:32.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 -- 172.21.15.38:0/22357481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd324074aa0 msgr2=0x7fd3241a4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:32.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd324074aa0 0x7fd3241a4c00 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd31800c9b0 tx=0x7fd31800ccc0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 -- 172.21.15.38:0/22357481 shutdown_connections 2024-03-26T06:36:32.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd324112cd0 0x7fd32419e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd30c0a29a0 0x7fd30c0a4e60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd324074180 0x7fd3241a46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 --2- 172.21.15.38:0/22357481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd324074aa0 0x7fd3241a4c00 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:32.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 -- 172.21.15.38:0/22357481 >> 172.21.15.38:0/22357481 conn(0x7fd3240fd7d0 msgr2=0x7fd3240fe440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:32.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 -- 172.21.15.38:0/22357481 shutdown_connections 2024-03-26T06:36:32.804 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:32.801+0000 7fd3295a1700 1 -- 172.21.15.38:0/22357481 wait complete. 2024-03-26T06:36:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.015+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/700080285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4ea41a5250 con 0x7f4ea4101680 2024-03-26T06:36:33.018 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.016+0000 7f4ea1ffb700 1 -- 172.21.15.38:0/700080285 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e9009cf20 con 0x7f4ea4101680 2024-03-26T06:36:33.019 INFO:teuthology.orchestra.run.smithi038.stdout:34359738449 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/700080285 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4e940a2920 msgr2=0x7f4e940a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4e940a2920 0x7f4e940a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009b10 tx=0x7f4e98009260 comp rx=0 tx=0).stop 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/700080285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ea4101680 msgr2=0x7f4ea4198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ea4101680 0x7f4ea4198a40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4e90000c00 tx=0x7f4e9000bb60 comp rx=0 tx=0).stop 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/700080285 shutdown_connections 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4e940a2920 0x7f4e940a4de0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ea4106d20 0x7f4ea419cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4ea4100ce0 0x7f4ea4198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 --2- 172.21.15.38:0/700080285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ea4101680 0x7f4ea4198a40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/700080285 >> 172.21.15.38:0/700080285 conn(0x7f4ea4077c10 msgr2=0x7f4ea4104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/700080285 shutdown_connections 2024-03-26T06:36:33.021 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.018+0000 7f4eaaaa1700 1 -- 172.21.15.38:0/700080285 wait complete. 2024-03-26T06:36:33.249 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:33.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:33 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/22357481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-26T06:36:33.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:33 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/700080285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T06:36:33.618 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:33 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/22357481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-26T06:36:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:33 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/700080285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T06:36:33.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:33 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/22357481' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-26T06:36:33.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:33 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/700080285' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T06:36:34.184 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350998 got 347892351002 for osd.11 2024-03-26T06:36:34.184 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:34.410 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:34 smithi038 ceph-mon[32265]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 7.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:34.410 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:34 smithi038 ceph-mon[32265]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T06:36:34.410 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:34 smithi038 ceph-mon[32265]: Cluster is now healthy 2024-03-26T06:36:34.410 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:34 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1461024101' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-26T06:36:34.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.866+0000 7f8884e16700 1 -- 172.21.15.38:0/1545665340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8880107e10 msgr2=0x7f88801081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:34.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.866+0000 7f8884e16700 1 --2- 172.21.15.38:0/1545665340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8880107e10 0x7f88801081f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f88700099c0 tx=0x7f8870009cd0 comp rx=0 tx=0).stop 2024-03-26T06:36:34.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.866+0000 7f8884e16700 1 -- 172.21.15.38:0/1545665340 shutdown_connections 2024-03-26T06:36:34.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.866+0000 7f8884e16700 1 --2- 172.21.15.38:0/1545665340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f88801027f0 0x7f888010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:34.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.866+0000 7f8884e16700 1 --2- 172.21.15.38:0/1545665340 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8880107e10 0x7f88801081f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.866+0000 7f8884e16700 1 --2- 172.21.15.38:0/1545665340 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8880101e30 0x7f88801022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.866+0000 7f8884e16700 1 -- 172.21.15.38:0/1545665340 >> 172.21.15.38:0/1545665340 conn(0x7f8880078f10 msgr2=0x7f8880079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.866+0000 7f8884e16700 1 -- 172.21.15.38:0/1545665340 shutdown_connections 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.866+0000 7f8884e16700 1 -- 172.21.15.38:0/1545665340 wait complete. 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.867+0000 7f8884e16700 1 Processor -- start 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.867+0000 7f8884e16700 1 -- start start 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.867+0000 7f8884e16700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8880101e30 0x7f88800740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.867+0000 7f8884e16700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88801027f0 0x7f8880074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.867+0000 7f8884e16700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8880107e10 0x7f8880074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.867+0000 7f8884e16700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8880112190 con 0x7f88801027f0 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.867+0000 7f8884e16700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8880112010 con 0x7f8880101e30 2024-03-26T06:36:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.867+0000 7f8884e16700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8880112310 con 0x7f8880107e10 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.867+0000 7f887dd9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88801027f0 0x7f8880074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.867+0000 7f887dd9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88801027f0 0x7f8880074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37372/0 (socket says 172.21.15.38:37372) 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.867+0000 7f887dd9b700 1 -- 172.21.15.38:0/1461024101 learned_addr learned my addr 172.21.15.38:0/1461024101 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f887e59c700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8880101e30 0x7f88800740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f887dd9b700 1 -- 172.21.15.38:0/1461024101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8880107e10 msgr2=0x7f8880074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f887ed9d700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8880107e10 0x7f8880074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f887dd9b700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8880107e10 0x7f8880074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f887dd9b700 1 -- 172.21.15.38:0/1461024101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8880101e30 msgr2=0x7f88800740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f887dd9b700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8880101e30 0x7f88800740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f887dd9b700 1 -- 172.21.15.38:0/1461024101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88801a6b90 con 0x7f88801027f0 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f887e59c700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8880101e30 0x7f88800740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f887ed9d700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8880107e10 0x7f8880074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:34.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f887dd9b700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88801027f0 0x7f8880074620 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f887400b530 tx=0x7f887400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f886f7fe700 1 -- 172.21.15.38:0/1461024101 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874015020 con 0x7f88801027f0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f886f7fe700 1 -- 172.21.15.38:0/1461024101 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8874011d10 con 0x7f88801027f0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f8884e16700 1 -- 172.21.15.38:0/1461024101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88801a6e80 con 0x7f88801027f0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f886f7fe700 1 -- 172.21.15.38:0/1461024101 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874010e50 con 0x7f88801027f0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.868+0000 7f8884e16700 1 -- 172.21.15.38:0/1461024101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88801a7330 con 0x7f88801027f0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.869+0000 7f8884e16700 1 -- 172.21.15.38:0/1461024101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8880068aa0 con 0x7f88801027f0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.870+0000 7f886f7fe700 1 -- 172.21.15.38:0/1461024101 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8874018020 con 0x7f88801027f0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.871+0000 7f886f7fe700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88680a2ac0 0x7f88680a4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.871+0000 7f887e59c700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88680a2ac0 0x7f88680a4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.871+0000 7f886f7fe700 1 -- 172.21.15.38:0/1461024101 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f88740ce8a0 con 0x7f88801027f0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.871+0000 7f887e59c700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88680a2ac0 0x7f88680a4f80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8870009750 tx=0x7f8870009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:33.875+0000 7f886f7fe700 1 -- 172.21.15.38:0/1461024101 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f887401f020 con 0x7f88801027f0 2024-03-26T06:36:34.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.082+0000 7f8884e16700 1 -- 172.21.15.38:0/1461024101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f888002d8d0 con 0x7f88801027f0 2024-03-26T06:36:34.505 INFO:teuthology.orchestra.run.smithi038.stdout:257698037801 2024-03-26T06:36:34.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.083+0000 7f886f7fe700 1 -- 172.21.15.38:0/1461024101 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f887409b8b0 con 0x7f88801027f0 2024-03-26T06:36:34.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 -- 172.21.15.38:0/1461024101 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88680a2ac0 msgr2=0x7f88680a4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:34.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88680a2ac0 0x7f88680a4f80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8870009750 tx=0x7f8870009f10 comp rx=0 tx=0).stop 2024-03-26T06:36:34.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 -- 172.21.15.38:0/1461024101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88801027f0 msgr2=0x7f8880074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:34.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88801027f0 0x7f8880074620 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f887400b530 tx=0x7f887400b8f0 comp rx=0 tx=0).stop 2024-03-26T06:36:34.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 -- 172.21.15.38:0/1461024101 shutdown_connections 2024-03-26T06:36:34.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f8880107e10 0x7f8880074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:34.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f88680a2ac0 0x7f88680a4f80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:34.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8880101e30 0x7f88800740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:34.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 --2- 172.21.15.38:0/1461024101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f88801027f0 0x7f8880074620 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:34.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 -- 172.21.15.38:0/1461024101 >> 172.21.15.38:0/1461024101 conn(0x7f8880078f10 msgr2=0x7f8880100ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:34.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 -- 172.21.15.38:0/1461024101 shutdown_connections 2024-03-26T06:36:34.506 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:34.085+0000 7f8884e16700 1 -- 172.21.15.38:0/1461024101 wait complete. 2024-03-26T06:36:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:34 smithi121 ceph-mon[36349]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 7.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:34 smithi121 ceph-mon[36349]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T06:36:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:34 smithi121 ceph-mon[36349]: Cluster is now healthy 2024-03-26T06:36:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:34 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1461024101' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-26T06:36:34.675 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:34 smithi170 ceph-mon[36912]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s; 7.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:34.675 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:34 smithi170 ceph-mon[36912]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T06:36:34.675 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:34 smithi170 ceph-mon[36912]: Cluster is now healthy 2024-03-26T06:36:34.675 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:34 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1461024101' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-26T06:36:35.097 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037795 got 257698037801 for osd.8 2024-03-26T06:36:35.097 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:35.667 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738445 got 34359738449 for osd.0 2024-03-26T06:36:35.667 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:36 smithi038 ceph-mon[32265]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1 op/s; 7.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:36.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:36 smithi121 ceph-mon[36349]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1 op/s; 7.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:36.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:36 smithi170 ceph-mon[36912]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1 op/s; 7.3 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:36.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.677+0000 7fccfd8c1700 1 -- 172.21.15.38:0/716973812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccf8105d10 msgr2=0x7fccf81060f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:36.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.677+0000 7fccfd8c1700 1 --2- 172.21.15.38:0/716973812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccf8105d10 0x7fccf81060f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcce80099c0 tx=0x7fcce8009cd0 comp rx=0 tx=0).stop 2024-03-26T06:36:36.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.677+0000 7fccfd8c1700 1 -- 172.21.15.38:0/716973812 shutdown_connections 2024-03-26T06:36:36.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.677+0000 7fccfd8c1700 1 --2- 172.21.15.38:0/716973812 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccf8102250 0x7fccf810dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:36.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.677+0000 7fccfd8c1700 1 --2- 172.21.15.38:0/716973812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccf8105d10 0x7fccf81060f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:36.681 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.677+0000 7fccfd8c1700 1 --2- 172.21.15.38:0/716973812 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf8101890 0x7fccf8101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:36.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.677+0000 7fccfd8c1700 1 -- 172.21.15.38:0/716973812 >> 172.21.15.38:0/716973812 conn(0x7fccf8076060 msgr2=0x7fccf8076470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:36.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.677+0000 7fccfd8c1700 1 -- 172.21.15.38:0/716973812 shutdown_connections 2024-03-26T06:36:36.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.677+0000 7fccfd8c1700 1 -- 172.21.15.38:0/716973812 wait complete. 2024-03-26T06:36:36.682 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.678+0000 7fccfd8c1700 1 Processor -- start 2024-03-26T06:36:36.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.678+0000 7fccfd8c1700 1 -- start start 2024-03-26T06:36:36.683 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.678+0000 7fccfd8c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf8101890 0x7fccf8102a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.678+0000 7fccfd8c1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccf8102250 0x7fccf8102f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.678+0000 7fccfd8c1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccf8105d10 0x7fccf8103490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.678+0000 7fccfd8c1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf8111090 con 0x7fccf8105d10 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.678+0000 7fccfd8c1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf8110f10 con 0x7fccf8101890 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.678+0000 7fccfd8c1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccf8111210 con 0x7fccf8102250 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.678+0000 7fccf77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccf8105d10 0x7fccf8103490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccf77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccf8105d10 0x7fccf8103490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37466/0 (socket says 172.21.15.38:37466) 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccf77fe700 1 -- 172.21.15.38:0/856399286 learned_addr learned my addr 172.21.15.38:0/856399286 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccf6ffd700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf8101890 0x7fccf8102a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccf67fc700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccf8102250 0x7fccf8102f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccf77fe700 1 -- 172.21.15.38:0/856399286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccf8102250 msgr2=0x7fccf8102f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:36.684 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccf77fe700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccf8102250 0x7fccf8102f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:36.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccf77fe700 1 -- 172.21.15.38:0/856399286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf8101890 msgr2=0x7fccf8102a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:36.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccf77fe700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf8101890 0x7fccf8102a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:36.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccf77fe700 1 -- 172.21.15.38:0/856399286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccf81a6af0 con 0x7fccf8105d10 2024-03-26T06:36:36.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccf77fe700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccf8105d10 0x7fccf8103490 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fccf000b970 tx=0x7fccf000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:36.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccfc8bf700 1 -- 172.21.15.38:0/856399286 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf000d860 con 0x7fccf8105d10 2024-03-26T06:36:36.685 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccfc8bf700 1 -- 172.21.15.38:0/856399286 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fccf000e430 con 0x7fccf8105d10 2024-03-26T06:36:36.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccfc8bf700 1 -- 172.21.15.38:0/856399286 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf0017660 con 0x7fccf8105d10 2024-03-26T06:36:36.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccfd8c1700 1 -- 172.21.15.38:0/856399286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccf81a6de0 con 0x7fccf8105d10 2024-03-26T06:36:36.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.679+0000 7fccfd8c1700 1 -- 172.21.15.38:0/856399286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccf81a7290 con 0x7fccf8105d10 2024-03-26T06:36:36.686 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.680+0000 7fccfd8c1700 1 -- 172.21.15.38:0/856399286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccf8068aa0 con 0x7fccf8105d10 2024-03-26T06:36:36.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.681+0000 7fccfc8bf700 1 -- 172.21.15.38:0/856399286 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fccf000e5e0 con 0x7fccf8105d10 2024-03-26T06:36:36.687 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.682+0000 7fccfc8bf700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcce00a2920 0x7fcce00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:36.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.682+0000 7fccf6ffd700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcce00a2920 0x7fcce00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:36.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.682+0000 7fccfc8bf700 1 -- 172.21.15.38:0/856399286 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fccf00ceb60 con 0x7fccf8105d10 2024-03-26T06:36:36.688 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.683+0000 7fccf6ffd700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcce00a2920 0x7fcce00a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009990 tx=0x7fcce8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:36.689 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.686+0000 7fccfc8bf700 1 -- 172.21.15.38:0/856399286 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fccf0013030 con 0x7fccf8105d10 2024-03-26T06:36:36.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.895+0000 7fccfd8c1700 1 -- 172.21.15.38:0/856399286 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fccf802d830 con 0x7fccf8105d10 2024-03-26T06:36:36.899 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.896+0000 7fccfc8bf700 1 -- 172.21.15.38:0/856399286 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fccf009ba60 con 0x7fccf8105d10 2024-03-26T06:36:36.900 INFO:teuthology.orchestra.run.smithi038.stdout:111669149763 2024-03-26T06:36:36.901 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 -- 172.21.15.38:0/856399286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcce00a2920 msgr2=0x7fcce00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcce00a2920 0x7fcce00a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009990 tx=0x7fcce8005fb0 comp rx=0 tx=0).stop 2024-03-26T06:36:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 -- 172.21.15.38:0/856399286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccf8105d10 msgr2=0x7fccf8103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccf8105d10 0x7fccf8103490 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fccf000b970 tx=0x7fccf000bc80 comp rx=0 tx=0).stop 2024-03-26T06:36:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 -- 172.21.15.38:0/856399286 shutdown_connections 2024-03-26T06:36:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fccf8105d10 0x7fccf8103490 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcce00a2920 0x7fcce00a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccf8101890 0x7fccf8102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 --2- 172.21.15.38:0/856399286 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fccf8102250 0x7fccf8102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 -- 172.21.15.38:0/856399286 >> 172.21.15.38:0/856399286 conn(0x7fccf8076060 msgr2=0x7fccf8100880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 -- 172.21.15.38:0/856399286 shutdown_connections 2024-03-26T06:36:36.902 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:36.898+0000 7fccfd8c1700 1 -- 172.21.15.38:0/856399286 wait complete. 2024-03-26T06:36:37.568 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:37 smithi170 ceph-mon[36912]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s; 6.0 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:37.569 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:37 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/856399286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T06:36:37.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:37 smithi038 ceph-mon[32265]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s; 6.0 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:37.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:37 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/856399286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T06:36:37.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:37 smithi121 ceph-mon[36349]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 2 op/s; 6.0 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:37.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:37 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/856399286' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T06:36:37.918 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149757 got 111669149763 for osd.3 2024-03-26T06:36:37.918 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:39.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.551+0000 7f9afd3b6700 1 -- 172.21.15.38:0/147910180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af8101e30 msgr2=0x7f9af81022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:39.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.551+0000 7f9afd3b6700 1 --2- 172.21.15.38:0/147910180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af8101e30 0x7f9af81022b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9ae00099c0 tx=0x7f9ae0009cd0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.556 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:39 smithi038 ceph-mon[32265]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 5.5 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:39.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.551+0000 7f9afd3b6700 1 -- 172.21.15.38:0/147910180 shutdown_connections 2024-03-26T06:36:39.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.551+0000 7f9afd3b6700 1 --2- 172.21.15.38:0/147910180 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9af81027f0 0x7f9af810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.551+0000 7f9afd3b6700 1 --2- 172.21.15.38:0/147910180 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9af8107e10 0x7f9af81081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.551+0000 7f9afd3b6700 1 --2- 172.21.15.38:0/147910180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af8101e30 0x7f9af81022b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.551+0000 7f9afd3b6700 1 -- 172.21.15.38:0/147910180 >> 172.21.15.38:0/147910180 conn(0x7f9af8078f10 msgr2=0x7f9af8079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:39.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.551+0000 7f9afd3b6700 1 -- 172.21.15.38:0/147910180 shutdown_connections 2024-03-26T06:36:39.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.551+0000 7f9afd3b6700 1 -- 172.21.15.38:0/147910180 wait complete. 2024-03-26T06:36:39.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.552+0000 7f9afd3b6700 1 Processor -- start 2024-03-26T06:36:39.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.552+0000 7f9afd3b6700 1 -- start start 2024-03-26T06:36:39.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.552+0000 7f9afd3b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9af8101e30 0x7f9af819a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:39.558 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.552+0000 7f9afd3b6700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9af81027f0 0x7f9af819ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.552+0000 7f9afd3b6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af8107e10 0x7f9af819ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.552+0000 7f9afd3b6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af81120c0 con 0x7f9af8107e10 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.552+0000 7f9afd3b6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af8111f40 con 0x7f9af8101e30 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.552+0000 7f9afd3b6700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af8112240 con 0x7f9af81027f0 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.552+0000 7f9af77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af8107e10 0x7f9af819ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.552+0000 7f9af77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af8107e10 0x7f9af819ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37482/0 (socket says 172.21.15.38:37482) 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.552+0000 7f9af77fe700 1 -- 172.21.15.38:0/1462851755 learned_addr learned my addr 172.21.15.38:0/1462851755 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9af67fc700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9af81027f0 0x7f9af819ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9af6ffd700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9af8101e30 0x7f9af819a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9af77fe700 1 -- 172.21.15.38:0/1462851755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9af81027f0 msgr2=0x7f9af819ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9af77fe700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9af81027f0 0x7f9af819ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9af77fe700 1 -- 172.21.15.38:0/1462851755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9af8101e30 msgr2=0x7f9af819a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:39.559 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9af77fe700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9af8101e30 0x7f9af819a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9af77fe700 1 -- 172.21.15.38:0/1462851755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af819f640 con 0x7f9af8107e10 2024-03-26T06:36:39.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9af77fe700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af8107e10 0x7f9af819ef00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9af000ada0 tx=0x7f9af000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:39.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9aeffff700 1 -- 172.21.15.38:0/1462851755 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af0017020 con 0x7f9af8107e10 2024-03-26T06:36:39.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9afd3b6700 1 -- 172.21.15.38:0/1462851755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af819f930 con 0x7f9af8107e10 2024-03-26T06:36:39.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9aeffff700 1 -- 172.21.15.38:0/1462851755 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9af000a0d0 con 0x7f9af8107e10 2024-03-26T06:36:39.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9aeffff700 1 -- 172.21.15.38:0/1462851755 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9af0005480 con 0x7f9af8107e10 2024-03-26T06:36:39.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.553+0000 7f9afd3b6700 1 -- 172.21.15.38:0/1462851755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af81a71b0 con 0x7f9af8107e10 2024-03-26T06:36:39.560 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.554+0000 7f9afd3b6700 1 -- 172.21.15.38:0/1462851755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9af8068aa0 con 0x7f9af8107e10 2024-03-26T06:36:39.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.555+0000 7f9aeffff700 1 -- 172.21.15.38:0/1462851755 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9af0005620 con 0x7f9af8107e10 2024-03-26T06:36:39.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.556+0000 7f9aeffff700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ae40a2920 0x7f9ae40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:39.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.556+0000 7f9af6ffd700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ae40a2920 0x7f9ae40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:39.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.556+0000 7f9aeffff700 1 -- 172.21.15.38:0/1462851755 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9af00ce340 con 0x7f9af8107e10 2024-03-26T06:36:39.561 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.557+0000 7f9af6ffd700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ae40a2920 0x7f9ae40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8005ea0 tx=0x7f9ae8005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:39.564 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.561+0000 7f9aeffff700 1 -- 172.21.15.38:0/1462851755 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9af0021020 con 0x7f9af8107e10 2024-03-26T06:36:39.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:39 smithi121 ceph-mon[36349]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 5.5 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:39.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:39 smithi170 ceph-mon[36912]: pgmap v324: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 3 op/s; 5.5 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:39.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.764+0000 7f9afd3b6700 1 -- 172.21.15.38:0/1462851755 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f9af81a7380 con 0x7f9af8107e10 2024-03-26T06:36:39.768 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.765+0000 7f9aeffff700 1 -- 172.21.15.38:0/1462851755 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9af009b310 con 0x7f9af8107e10 2024-03-26T06:36:39.769 INFO:teuthology.orchestra.run.smithi038.stdout:146028888127 2024-03-26T06:36:39.770 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 -- 172.21.15.38:0/1462851755 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ae40a2920 msgr2=0x7f9ae40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:39.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ae40a2920 0x7f9ae40a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9ae8005ea0 tx=0x7f9ae8005e30 comp rx=0 tx=0).stop 2024-03-26T06:36:39.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 -- 172.21.15.38:0/1462851755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af8107e10 msgr2=0x7f9af819ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:39.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af8107e10 0x7f9af819ef00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9af000ada0 tx=0x7f9af000c330 comp rx=0 tx=0).stop 2024-03-26T06:36:39.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 -- 172.21.15.38:0/1462851755 shutdown_connections 2024-03-26T06:36:39.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af8107e10 0x7f9af819ef00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9ae40a2920 0x7f9ae40a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9af8101e30 0x7f9af819a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 --2- 172.21.15.38:0/1462851755 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9af81027f0 0x7f9af819ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 -- 172.21.15.38:0/1462851755 >> 172.21.15.38:0/1462851755 conn(0x7f9af8078f10 msgr2=0x7f9af8074240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:39.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 -- 172.21.15.38:0/1462851755 shutdown_connections 2024-03-26T06:36:39.771 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.767+0000 7f9afd3b6700 1 -- 172.21.15.38:0/1462851755 wait complete. 2024-03-26T06:36:39.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.778+0000 7f636a584700 1 -- 172.21.15.38:0/3890107871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6364101e30 msgr2=0x7f63641022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:39.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.778+0000 7f636a584700 1 --2- 172.21.15.38:0/3890107871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6364101e30 0x7f63641022b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f63540099c0 tx=0x7f6354009cd0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.778+0000 7f636a584700 1 -- 172.21.15.38:0/3890107871 shutdown_connections 2024-03-26T06:36:39.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.778+0000 7f636a584700 1 --2- 172.21.15.38:0/3890107871 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63641027f0 0x7f636410ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.778+0000 7f636a584700 1 --2- 172.21.15.38:0/3890107871 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6364107e10 0x7f63641081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.781 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.778+0000 7f636a584700 1 --2- 172.21.15.38:0/3890107871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6364101e30 0x7f63641022b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.778+0000 7f636a584700 1 -- 172.21.15.38:0/3890107871 >> 172.21.15.38:0/3890107871 conn(0x7f6364078f10 msgr2=0x7f6364079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:39.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.778+0000 7f636a584700 1 -- 172.21.15.38:0/3890107871 shutdown_connections 2024-03-26T06:36:39.782 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.778+0000 7f636a584700 1 -- 172.21.15.38:0/3890107871 wait complete. 2024-03-26T06:36:39.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.778+0000 7f636a584700 1 Processor -- start 2024-03-26T06:36:39.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.778+0000 7f636a584700 1 -- start start 2024-03-26T06:36:39.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f636a584700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6364101e30 0x7f63641984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:39.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f636a584700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63641027f0 0x7f6364198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:39.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f636a584700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6364107e10 0x7f636419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:39.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f636a584700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6364112090 con 0x7f6364107e10 2024-03-26T06:36:39.784 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f636a584700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6364111f10 con 0x7f6364101e30 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f636a584700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6364112210 con 0x7f63641027f0 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f6368b21700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6364107e10 0x7f636419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f6368b21700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6364107e10 0x7f636419cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37512/0 (socket says 172.21.15.38:37512) 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f6368b21700 1 -- 172.21.15.38:0/2567418021 learned_addr learned my addr 172.21.15.38:0/2567418021 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f63637fe700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63641027f0 0x7f6364198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f6363fff700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6364101e30 0x7f63641984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f6368b21700 1 -- 172.21.15.38:0/2567418021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63641027f0 msgr2=0x7f6364198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f6368b21700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63641027f0 0x7f6364198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f6368b21700 1 -- 172.21.15.38:0/2567418021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6364101e30 msgr2=0x7f63641984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f6368b21700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6364101e30 0x7f63641984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f6368b21700 1 -- 172.21.15.38:0/2567418021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636419d4d0 con 0x7f6364107e10 2024-03-26T06:36:39.785 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.779+0000 7f63637fe700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63641027f0 0x7f6364198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:36:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.780+0000 7f6368b21700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6364107e10 0x7f636419cd90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f635c00ada0 tx=0x7f635c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.780+0000 7f63617fa700 1 -- 172.21.15.38:0/2567418021 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c017020 con 0x7f6364107e10 2024-03-26T06:36:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.780+0000 7f636a584700 1 -- 172.21.15.38:0/2567418021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f636419d7c0 con 0x7f6364107e10 2024-03-26T06:36:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.780+0000 7f63617fa700 1 -- 172.21.15.38:0/2567418021 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f635c00cdb0 con 0x7f6364107e10 2024-03-26T06:36:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.780+0000 7f63617fa700 1 -- 172.21.15.38:0/2567418021 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c0053e0 con 0x7f6364107e10 2024-03-26T06:36:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.780+0000 7f636a584700 1 -- 172.21.15.38:0/2567418021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63641a5040 con 0x7f6364107e10 2024-03-26T06:36:39.786 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.781+0000 7f636a584700 1 -- 172.21.15.38:0/2567418021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6364068aa0 con 0x7f6364107e10 2024-03-26T06:36:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.782+0000 7f63617fa700 1 -- 172.21.15.38:0/2567418021 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f635c00c8d0 con 0x7f6364107e10 2024-03-26T06:36:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.783+0000 7f63617fa700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f63500a29a0 0x7f63500a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.783+0000 7f6363fff700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f63500a29a0 0x7f63500a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.783+0000 7f63617fa700 1 -- 172.21.15.38:0/2567418021 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f635c0ce8d0 con 0x7f6364107e10 2024-03-26T06:36:39.787 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.783+0000 7f6363fff700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f63500a29a0 0x7f63500a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f634c005950 tx=0x7f634c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:39.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.787+0000 7f63617fa700 1 -- 172.21.15.38:0/2567418021 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f635c021020 con 0x7f6364107e10 2024-03-26T06:36:39.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.994+0000 7f636a584700 1 -- 172.21.15.38:0/2567418021 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f636402d830 con 0x7f6364107e10 2024-03-26T06:36:39.998 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.994+0000 7f63617fa700 1 -- 172.21.15.38:0/2567418021 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f635c09b8a0 con 0x7f6364107e10 2024-03-26T06:36:39.998 INFO:teuthology.orchestra.run.smithi038.stdout:313532612640 2024-03-26T06:36:40.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 -- 172.21.15.38:0/2567418021 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f63500a29a0 msgr2=0x7f63500a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:40.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f63500a29a0 0x7f63500a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f634c005950 tx=0x7f634c00a300 comp rx=0 tx=0).stop 2024-03-26T06:36:40.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 -- 172.21.15.38:0/2567418021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6364107e10 msgr2=0x7f636419cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:40.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6364107e10 0x7f636419cd90 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f635c00ada0 tx=0x7f635c00c330 comp rx=0 tx=0).stop 2024-03-26T06:36:40.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 -- 172.21.15.38:0/2567418021 shutdown_connections 2024-03-26T06:36:40.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6364107e10 0x7f636419cd90 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:40.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f63500a29a0 0x7f63500a4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:40.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6364101e30 0x7f63641984c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:40.000 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 --2- 172.21.15.38:0/2567418021 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f63641027f0 0x7f6364198a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:40.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 -- 172.21.15.38:0/2567418021 >> 172.21.15.38:0/2567418021 conn(0x7f6364078f10 msgr2=0x7f6364100ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:40.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 -- 172.21.15.38:0/2567418021 shutdown_connections 2024-03-26T06:36:40.001 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:39.996+0000 7f636a584700 1 -- 172.21.15.38:0/2567418021 wait complete. 2024-03-26T06:36:40.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:40 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1462851755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T06:36:40.544 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:40 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2567418021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-26T06:36:40.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:40 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1462851755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T06:36:40.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:40 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2567418021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-26T06:36:40.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:40 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1462851755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T06:36:40.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:40 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2567418021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-26T06:36:40.843 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612634 got 313532612640 for osd.10 2024-03-26T06:36:40.843 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:41.582 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:41 smithi038 ceph-mon[32265]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3 op/s; 4.9 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:41.617 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888122 got 146028888127 for osd.4 2024-03-26T06:36:41.617 DEBUG:teuthology.parallel:result is None 2024-03-26T06:36:41.617 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T06:36:41.617 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T06:36:41.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:41 smithi121 ceph-mon[36349]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3 op/s; 4.9 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:41.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:41 smithi170 ceph-mon[36912]: pgmap v325: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 3 op/s; 4.9 KiB/s, 0 keys/s, 1 objects/s recovering 2024-03-26T06:36:41.895 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:43.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:43.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:43 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:43.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.609+0000 7f62b0086700 1 -- 172.21.15.38:0/2111730604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8101680 msgr2=0x7f62a810db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:43.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.609+0000 7f62b0086700 1 --2- 172.21.15.38:0/2111730604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8101680 0x7f62a810db60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f62a000b0a0 tx=0x7f62a000b3b0 comp rx=0 tx=0).stop 2024-03-26T06:36:43.614 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.609+0000 7f62b0086700 1 -- 172.21.15.38:0/2111730604 shutdown_connections 2024-03-26T06:36:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.609+0000 7f62b0086700 1 --2- 172.21.15.38:0/2111730604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8101680 0x7f62a810db60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.609+0000 7f62b0086700 1 --2- 172.21.15.38:0/2111730604 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a8106d20 0x7f62a8107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.609+0000 7f62b0086700 1 --2- 172.21.15.38:0/2111730604 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a8100ce0 0x7f62a8101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.609+0000 7f62b0086700 1 -- 172.21.15.38:0/2111730604 >> 172.21.15.38:0/2111730604 conn(0x7f62a8077c10 msgr2=0x7f62a8078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.609+0000 7f62b0086700 1 -- 172.21.15.38:0/2111730604 shutdown_connections 2024-03-26T06:36:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.609+0000 7f62b0086700 1 -- 172.21.15.38:0/2111730604 wait complete. 2024-03-26T06:36:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.610+0000 7f62b0086700 1 Processor -- start 2024-03-26T06:36:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.610+0000 7f62b0086700 1 -- start start 2024-03-26T06:36:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.610+0000 7f62b0086700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a8100ce0 0x7f62a8199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.610+0000 7f62b0086700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8101680 0x7f62a81998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:43.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62b0086700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a8106d20 0x7f62a81a4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:43.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62b0086700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a8111010 con 0x7f62a8101680 2024-03-26T06:36:43.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62b0086700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a8110e90 con 0x7f62a8100ce0 2024-03-26T06:36:43.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62b0086700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a8111190 con 0x7f62a8106d20 2024-03-26T06:36:43.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62ad621700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8101680 0x7f62a81998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:43.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62ad621700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8101680 0x7f62a81998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37538/0 (socket says 172.21.15.38:37538) 2024-03-26T06:36:43.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62ad621700 1 -- 172.21.15.38:0/194244826 learned_addr learned my addr 172.21.15.38:0/194244826 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:43.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62ade22700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a8100ce0 0x7f62a8199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:43.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62ae623700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a8106d20 0x7f62a81a4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:43.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62ad621700 1 -- 172.21.15.38:0/194244826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a8106d20 msgr2=0x7f62a81a4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:43.616 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62ad621700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a8106d20 0x7f62a81a4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:43.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62ad621700 1 -- 172.21.15.38:0/194244826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a8100ce0 msgr2=0x7f62a8199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:43.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62ad621700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a8100ce0 0x7f62a8199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:43.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62ad621700 1 -- 172.21.15.38:0/194244826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a81a4950 con 0x7f62a8101680 2024-03-26T06:36:43.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62ad621700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8101680 0x7f62a81998d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f62a400d980 tx=0x7f62a400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:43.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f629effd700 1 -- 172.21.15.38:0/194244826 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a4016070 con 0x7f62a8101680 2024-03-26T06:36:43.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62b0086700 1 -- 172.21.15.38:0/194244826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a81a4b80 con 0x7f62a8101680 2024-03-26T06:36:43.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f629effd700 1 -- 172.21.15.38:0/194244826 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62a4004d10 con 0x7f62a8101680 2024-03-26T06:36:43.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f629effd700 1 -- 172.21.15.38:0/194244826 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a40055f0 con 0x7f62a8101680 2024-03-26T06:36:43.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.611+0000 7f62b0086700 1 -- 172.21.15.38:0/194244826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a81a5110 con 0x7f62a8101680 2024-03-26T06:36:43.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.612+0000 7f62b0086700 1 -- 172.21.15.38:0/194244826 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a8068aa0 con 0x7f62a8101680 2024-03-26T06:36:43.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.614+0000 7f629effd700 1 -- 172.21.15.38:0/194244826 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f62a4004750 con 0x7f62a8101680 2024-03-26T06:36:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.614+0000 7f629effd700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62940a2920 0x7f62940a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.614+0000 7f62ade22700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62940a2920 0x7f62940a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.614+0000 7f629effd700 1 -- 172.21.15.38:0/194244826 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f62a40cef60 con 0x7f62a8101680 2024-03-26T06:36:43.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.615+0000 7f62ade22700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62940a2920 0x7f62940a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f62980045b0 tx=0x7f629800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:43.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.618+0000 7f629effd700 1 -- 172.21.15.38:0/194244826 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f62a401f020 con 0x7f62a8101680 2024-03-26T06:36:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:43 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:43.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:43 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:43.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.820+0000 7f62b0086700 1 -- 172.21.15.38:0/194244826 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62a8062220 con 0x7f62940a2920 2024-03-26T06:36:43.830 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.827+0000 7f629effd700 1 -- 172.21.15.38:0/194244826 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258726 (secure 0 0 0) 0x7f62a8062220 con 0x7f62940a2920 2024-03-26T06:36:43.834 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 -- 172.21.15.38:0/194244826 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62940a2920 msgr2=0x7f62940a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62940a2920 0x7f62940a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f62980045b0 tx=0x7f629800b3c0 comp rx=0 tx=0).stop 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 -- 172.21.15.38:0/194244826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8101680 msgr2=0x7f62a81998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8101680 0x7f62a81998d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f62a400d980 tx=0x7f62a400dc90 comp rx=0 tx=0).stop 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 -- 172.21.15.38:0/194244826 shutdown_connections 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a8106d20 0x7f62a81a4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f62940a2920 0x7f62940a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a8100ce0 0x7f62a8199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 --2- 172.21.15.38:0/194244826 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a8101680 0x7f62a81998d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 -- 172.21.15.38:0/194244826 >> 172.21.15.38:0/194244826 conn(0x7f62a8077c10 msgr2=0x7f62a80ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 -- 172.21.15.38:0/194244826 shutdown_connections 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:43.831+0000 7f62b0086700 1 -- 172.21.15.38:0/194244826 wait complete. 2024-03-26T06:36:43.837 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-26T06:36:44.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:44 smithi038 ceph-mon[32265]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s; 4.9 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-26T06:36:44.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:44 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:36:44.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:44 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:36:44.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:44 smithi038 ceph-mon[32265]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:44.510 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-03-26T06:36:42.251855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654068,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":550,"num_read_kb":393,"num_write":338,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1240783,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370912,"kb_used_data":13356,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570912,"statfs":{"total":1151940427776,"available":1151560613888,"internally_reserved":0,"allocated":13676544,"data_stored":7467432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":59,"num_read_kb":32,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":60047,"num_keys_recovered":7,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002961"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531887+0000","last_change":"2024-03-26T06:35:27.137165+0000","last_active":"2024-03-26T06:36:27.531887+0000","last_peered":"2024-03-26T06:36:27.531887+0000","last_clean":"2024-03-26T06:36:27.531887+0000","last_became_active":"2024-03-26T06:35:27.136869+0000","last_became_peered":"2024-03-26T06:35:27.136869+0000","last_unstale":"2024-03-26T06:36:27.531887+0000","last_undegraded":"2024-03-26T06:36:27.531887+0000","last_fullsized":"2024-03-26T06:36:27.531887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:43:36.671967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535676+0000","last_change":"2024-03-26T06:35:27.130619+0000","last_active":"2024-03-26T06:36:27.535676+0000","last_peered":"2024-03-26T06:36:27.535676+0000","last_clean":"2024-03-26T06:36:27.535676+0000","last_became_active":"2024-03-26T06:35:27.128846+0000","last_became_peered":"2024-03-26T06:35:27.128846+0000","last_unstale":"2024-03-26T06:36:27.535676+0000","last_undegraded":"2024-03-26T06:36:27.535676+0000","last_fullsized":"2024-03-26T06:36:27.535676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:14:15.627268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535568+0000","last_change":"2024-03-26T06:35:27.138214+0000","last_active":"2024-03-26T06:36:27.535568+0000","last_peered":"2024-03-26T06:36:27.535568+0000","last_clean":"2024-03-26T06:36:27.535568+0000","last_became_active":"2024-03-26T06:35:27.137846+0000","last_became_peered":"2024-03-26T06:35:27.137846+0000","last_unstale":"2024-03-26T06:36:27.535568+0000","last_undegraded":"2024-03-26T06:36:27.535568+0000","last_fullsized":"2024-03-26T06:36:27.535568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:48:46.441889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562934+0000","last_change":"2024-03-26T06:35:27.129582+0000","last_active":"2024-03-26T06:36:27.562934+0000","last_peered":"2024-03-26T06:36:27.562934+0000","last_clean":"2024-03-26T06:36:27.562934+0000","last_became_active":"2024-03-26T06:35:27.129245+0000","last_became_peered":"2024-03-26T06:35:27.129245+0000","last_unstale":"2024-03-26T06:36:27.562934+0000","last_undegraded":"2024-03-26T06:36:27.562934+0000","last_fullsized":"2024-03-26T06:36:27.562934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:09:14.548067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536655+0000","last_change":"2024-03-26T06:35:27.131993+0000","last_active":"2024-03-26T06:36:27.536655+0000","last_peered":"2024-03-26T06:36:27.536655+0000","last_clean":"2024-03-26T06:36:27.536655+0000","last_became_active":"2024-03-26T06:35:27.131650+0000","last_became_peered":"2024-03-26T06:35:27.131650+0000","last_unstale":"2024-03-26T06:36:27.536655+0000","last_undegraded":"2024-03-26T06:36:27.536655+0000","last_fullsized":"2024-03-26T06:36:27.536655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:38:41.407314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.532074+0000","last_change":"2024-03-26T06:35:27.130325+0000","last_active":"2024-03-26T06:36:27.532074+0000","last_peered":"2024-03-26T06:36:27.532074+0000","last_clean":"2024-03-26T06:36:27.532074+0000","last_became_active":"2024-03-26T06:35:27.129658+0000","last_became_peered":"2024-03-26T06:35:27.129658+0000","last_unstale":"2024-03-26T06:36:27.532074+0000","last_undegraded":"2024-03-26T06:36:27.532074+0000","last_fullsized":"2024-03-26T06:36:27.532074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:25:18.580839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535725+0000","last_change":"2024-03-26T06:35:27.130870+0000","last_active":"2024-03-26T06:36:27.535725+0000","last_peered":"2024-03-26T06:36:27.535725+0000","last_clean":"2024-03-26T06:36:27.535725+0000","last_became_active":"2024-03-26T06:35:27.130521+0000","last_became_peered":"2024-03-26T06:35:27.130521+0000","last_unstale":"2024-03-26T06:36:27.535725+0000","last_undegraded":"2024-03-26T06:36:27.535725+0000","last_fullsized":"2024-03-26T06:36:27.535725+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:58:40.640708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540027+0000","last_change":"2024-03-26T06:35:27.136258+0000","last_active":"2024-03-26T06:36:27.540027+0000","last_peered":"2024-03-26T06:36:27.540027+0000","last_clean":"2024-03-26T06:36:27.540027+0000","last_became_active":"2024-03-26T06:35:27.135746+0000","last_became_peered":"2024-03-26T06:35:27.135746+0000","last_unstale":"2024-03-26T06:36:27.540027+0000","last_undegraded":"2024-03-26T06:36:27.540027+0000","last_fullsized":"2024-03-26T06:36:27.540027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:42:24.270717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534322+0000","last_change":"2024-03-26T06:35:27.130533+0000","last_active":"2024-03-26T06:36:27.534322+0000","last_peered":"2024-03-26T06:36:27.534322+0000","last_clean":"2024-03-26T06:36:27.534322+0000","last_became_active":"2024-03-26T06:35:27.130251+0000","last_became_peered":"2024-03-26T06:35:27.130251+0000","last_unstale":"2024-03-26T06:36:27.534322+0000","last_undegraded":"2024-03-26T06:36:27.534322+0000","last_fullsized":"2024-03-26T06:36:27.534322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T18:13:40.209389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.539256+0000","last_change":"2024-03-26T06:36:27.539256+0000","last_active":"2024-03-26T06:36:27.539256+0000","last_peered":"2024-03-26T06:36:27.539256+0000","last_clean":"2024-03-26T06:36:27.539256+0000","last_became_active":"2024-03-26T06:36:27.538934+0000","last_became_peered":"2024-03-26T06:36:27.538934+0000","last_unstale":"2024-03-26T06:36:27.539256+0000","last_undegraded":"2024-03-26T06:36:27.539256+0000","last_fullsized":"2024-03-26T06:36:27.539256+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:29:04.582141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536010+0000","last_change":"2024-03-26T06:35:27.137285+0000","last_active":"2024-03-26T06:36:27.536010+0000","last_peered":"2024-03-26T06:36:27.536010+0000","last_clean":"2024-03-26T06:36:27.536010+0000","last_became_active":"2024-03-26T06:35:27.136930+0000","last_became_peered":"2024-03-26T06:35:27.136930+0000","last_unstale":"2024-03-26T06:36:27.536010+0000","last_undegraded":"2024-03-26T06:36:27.536010+0000","last_fullsized":"2024-03-26T06:36:27.536010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:36:42.036591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562901+0000","last_change":"2024-03-26T06:35:27.133279+0000","last_active":"2024-03-26T06:36:27.562901+0000","last_peered":"2024-03-26T06:36:27.562901+0000","last_clean":"2024-03-26T06:36:27.562901+0000","last_became_active":"2024-03-26T06:35:27.132860+0000","last_became_peered":"2024-03-26T06:35:27.132860+0000","last_unstale":"2024-03-26T06:36:27.562901+0000","last_undegraded":"2024-03-26T06:36:27.562901+0000","last_fullsized":"2024-03-26T06:36:27.562901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:36:34.455411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562469+0000","last_change":"2024-03-26T06:35:27.124143+0000","last_active":"2024-03-26T06:36:27.562469+0000","last_peered":"2024-03-26T06:36:27.562469+0000","last_clean":"2024-03-26T06:36:27.562469+0000","last_became_active":"2024-03-26T06:35:27.123774+0000","last_became_peered":"2024-03-26T06:35:27.123774+0000","last_unstale":"2024-03-26T06:36:27.562469+0000","last_undegraded":"2024-03-26T06:36:27.562469+0000","last_fullsized":"2024-03-26T06:36:27.562469+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:05:14.135531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535360+0000","last_change":"2024-03-26T06:35:27.127523+0000","last_active":"2024-03-26T06:36:27.535360+0000","last_peered":"2024-03-26T06:36:27.535360+0000","last_clean":"2024-03-26T06:36:27.535360+0000","last_became_active":"2024-03-26T06:35:27.127141+0000","last_became_peered":"2024-03-26T06:35:27.127141+0000","last_unstale":"2024-03-26T06:36:27.535360+0000","last_undegraded":"2024-03-26T06:36:27.535360+0000","last_fullsized":"2024-03-26T06:36:27.535360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:21:44.524721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.563452+0000","last_change":"2024-03-26T06:36:27.563452+0000","last_active":"2024-03-26T06:36:27.563452+0000","last_peered":"2024-03-26T06:36:27.563452+0000","last_clean":"2024-03-26T06:36:27.563452+0000","last_became_active":"2024-03-26T06:36:27.563070+0000","last_became_peered":"2024-03-26T06:36:27.563070+0000","last_unstale":"2024-03-26T06:36:27.563452+0000","last_undegraded":"2024-03-26T06:36:27.563452+0000","last_fullsized":"2024-03-26T06:36:27.563452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:14:41.816751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.533805+0000","last_change":"2024-03-26T06:35:27.132021+0000","last_active":"2024-03-26T06:36:27.533805+0000","last_peered":"2024-03-26T06:36:27.533805+0000","last_clean":"2024-03-26T06:36:27.533805+0000","last_became_active":"2024-03-26T06:35:27.131722+0000","last_became_peered":"2024-03-26T06:35:27.131722+0000","last_unstale":"2024-03-26T06:36:27.533805+0000","last_undegraded":"2024-03-26T06:36:27.533805+0000","last_fullsized":"2024-03-26T06:36:27.533805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:52:23.253250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531749+0000","last_change":"2024-03-26T06:35:27.137225+0000","last_active":"2024-03-26T06:36:27.531749+0000","last_peered":"2024-03-26T06:36:27.531749+0000","last_clean":"2024-03-26T06:36:27.531749+0000","last_became_active":"2024-03-26T06:35:27.136935+0000","last_became_peered":"2024-03-26T06:35:27.136935+0000","last_unstale":"2024-03-26T06:36:27.531749+0000","last_undegraded":"2024-03-26T06:36:27.531749+0000","last_fullsized":"2024-03-26T06:36:27.531749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:14:02.540482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.538699+0000","last_change":"2024-03-26T06:35:27.136068+0000","last_active":"2024-03-26T06:36:27.538699+0000","last_peered":"2024-03-26T06:36:27.538699+0000","last_clean":"2024-03-26T06:36:27.538699+0000","last_became_active":"2024-03-26T06:35:27.135586+0000","last_became_peered":"2024-03-26T06:35:27.135586+0000","last_unstale":"2024-03-26T06:36:27.538699+0000","last_undegraded":"2024-03-26T06:36:27.538699+0000","last_fullsized":"2024-03-26T06:36:27.538699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:21:51.729675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.563455+0000","last_change":"2024-03-26T06:36:27.563455+0000","last_active":"2024-03-26T06:36:27.563455+0000","last_peered":"2024-03-26T06:36:27.563455+0000","last_clean":"2024-03-26T06:36:27.563455+0000","last_became_active":"2024-03-26T06:36:27.563115+0000","last_became_peered":"2024-03-26T06:36:27.563115+0000","last_unstale":"2024-03-26T06:36:27.563455+0000","last_undegraded":"2024-03-26T06:36:27.563455+0000","last_fullsized":"2024-03-26T06:36:27.563455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:30:17.781959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.546513+0000","last_change":"2024-03-26T06:36:27.546513+0000","last_active":"2024-03-26T06:36:27.546513+0000","last_peered":"2024-03-26T06:36:27.546513+0000","last_clean":"2024-03-26T06:36:27.546513+0000","last_became_active":"2024-03-26T06:36:27.540060+0000","last_became_peered":"2024-03-26T06:36:27.540060+0000","last_unstale":"2024-03-26T06:36:27.546513+0000","last_undegraded":"2024-03-26T06:36:27.546513+0000","last_fullsized":"2024-03-26T06:36:27.546513+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:22:58.631595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540849+0000","last_change":"2024-03-26T06:35:27.141510+0000","last_active":"2024-03-26T06:36:27.540849+0000","last_peered":"2024-03-26T06:36:27.540849+0000","last_clean":"2024-03-26T06:36:27.540849+0000","last_became_active":"2024-03-26T06:35:27.141182+0000","last_became_peered":"2024-03-26T06:35:27.141182+0000","last_unstale":"2024-03-26T06:36:27.540849+0000","last_undegraded":"2024-03-26T06:36:27.540849+0000","last_fullsized":"2024-03-26T06:36:27.540849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:40:55.993939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.541017+0000","last_change":"2024-03-26T06:35:26.095502+0000","last_active":"2024-03-26T06:36:27.541017+0000","last_peered":"2024-03-26T06:36:27.541017+0000","last_clean":"2024-03-26T06:36:27.541017+0000","last_became_active":"2024-03-26T06:35:26.095168+0000","last_became_peered":"2024-03-26T06:35:26.095168+0000","last_unstale":"2024-03-26T06:36:27.541017+0000","last_undegraded":"2024-03-26T06:36:27.541017+0000","last_fullsized":"2024-03-26T06:36:27.541017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:01:44.307274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535881+0000","last_change":"2024-03-26T06:35:27.137179+0000","last_active":"2024-03-26T06:36:27.535881+0000","last_peered":"2024-03-26T06:36:27.535881+0000","last_clean":"2024-03-26T06:36:27.535881+0000","last_became_active":"2024-03-26T06:35:27.136660+0000","last_became_peered":"2024-03-26T06:35:27.136660+0000","last_unstale":"2024-03-26T06:36:27.535881+0000","last_undegraded":"2024-03-26T06:36:27.535881+0000","last_fullsized":"2024-03-26T06:36:27.535881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:16:16.042134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536655+0000","last_change":"2024-03-26T06:35:26.105395+0000","last_active":"2024-03-26T06:36:27.536655+0000","last_peered":"2024-03-26T06:36:27.536655+0000","last_clean":"2024-03-26T06:36:27.536655+0000","last_became_active":"2024-03-26T06:35:26.104879+0000","last_became_peered":"2024-03-26T06:35:26.104879+0000","last_unstale":"2024-03-26T06:36:27.536655+0000","last_undegraded":"2024-03-26T06:36:27.536655+0000","last_fullsized":"2024-03-26T06:36:27.536655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:44:57.444844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.533980+0000","last_change":"2024-03-26T06:35:27.133549+0000","last_active":"2024-03-26T06:36:27.533980+0000","last_peered":"2024-03-26T06:36:27.533980+0000","last_clean":"2024-03-26T06:36:27.533980+0000","last_became_active":"2024-03-26T06:35:27.132176+0000","last_became_peered":"2024-03-26T06:35:27.132176+0000","last_unstale":"2024-03-26T06:36:27.533980+0000","last_undegraded":"2024-03-26T06:36:27.533980+0000","last_fullsized":"2024-03-26T06:36:27.533980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:15:59.281268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536475+0000","last_change":"2024-03-26T06:35:26.101438+0000","last_active":"2024-03-26T06:36:27.536475+0000","last_peered":"2024-03-26T06:36:27.536475+0000","last_clean":"2024-03-26T06:36:27.536475+0000","last_became_active":"2024-03-26T06:35:26.101139+0000","last_became_peered":"2024-03-26T06:35:26.101139+0000","last_unstale":"2024-03-26T06:36:27.536475+0000","last_undegraded":"2024-03-26T06:36:27.536475+0000","last_fullsized":"2024-03-26T06:36:27.536475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:10:36.750046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535900+0000","last_change":"2024-03-26T06:35:27.127896+0000","last_active":"2024-03-26T06:36:27.535900+0000","last_peered":"2024-03-26T06:36:27.535900+0000","last_clean":"2024-03-26T06:36:27.535900+0000","last_became_active":"2024-03-26T06:35:27.127558+0000","last_became_peered":"2024-03-26T06:35:27.127558+0000","last_unstale":"2024-03-26T06:36:27.535900+0000","last_undegraded":"2024-03-26T06:36:27.535900+0000","last_fullsized":"2024-03-26T06:36:27.535900+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:04:58.033255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:38.274449+0000","last_change":"2024-03-26T06:35:26.100148+0000","last_active":"2024-03-26T06:36:38.274449+0000","last_peered":"2024-03-26T06:36:38.274449+0000","last_clean":"2024-03-26T06:36:38.274449+0000","last_became_active":"2024-03-26T06:35:26.099827+0000","last_became_peered":"2024-03-26T06:35:26.099827+0000","last_unstale":"2024-03-26T06:36:38.274449+0000","last_undegraded":"2024-03-26T06:36:38.274449+0000","last_fullsized":"2024-03-26T06:36:38.274449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:33:21.845866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536161+0000","last_change":"2024-03-26T06:35:27.125318+0000","last_active":"2024-03-26T06:36:27.536161+0000","last_peered":"2024-03-26T06:36:27.536161+0000","last_clean":"2024-03-26T06:36:27.536161+0000","last_became_active":"2024-03-26T06:35:27.125013+0000","last_became_peered":"2024-03-26T06:35:27.125013+0000","last_unstale":"2024-03-26T06:36:27.536161+0000","last_undegraded":"2024-03-26T06:36:27.536161+0000","last_fullsized":"2024-03-26T06:36:27.536161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:50:25.123317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.532043+0000","last_change":"2024-03-26T06:35:26.097150+0000","last_active":"2024-03-26T06:36:27.532043+0000","last_peered":"2024-03-26T06:36:27.532043+0000","last_clean":"2024-03-26T06:36:27.532043+0000","last_became_active":"2024-03-26T06:35:26.096844+0000","last_became_peered":"2024-03-26T06:35:26.096844+0000","last_unstale":"2024-03-26T06:36:27.532043+0000","last_undegraded":"2024-03-26T06:36:27.532043+0000","last_fullsized":"2024-03-26T06:36:27.532043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:57:47.399493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535756+0000","last_change":"2024-03-26T06:35:27.132261+0000","last_active":"2024-03-26T06:36:27.535756+0000","last_peered":"2024-03-26T06:36:27.535756+0000","last_clean":"2024-03-26T06:36:27.535756+0000","last_became_active":"2024-03-26T06:35:27.131893+0000","last_became_peered":"2024-03-26T06:35:27.131893+0000","last_unstale":"2024-03-26T06:36:27.535756+0000","last_undegraded":"2024-03-26T06:36:27.535756+0000","last_fullsized":"2024-03-26T06:36:27.535756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:42.521515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562425+0000","last_change":"2024-03-26T06:35:26.092356+0000","last_active":"2024-03-26T06:36:27.562425+0000","last_peered":"2024-03-26T06:36:27.562425+0000","last_clean":"2024-03-26T06:36:27.562425+0000","last_became_active":"2024-03-26T06:35:26.091948+0000","last_became_peered":"2024-03-26T06:35:26.091948+0000","last_unstale":"2024-03-26T06:36:27.562425+0000","last_undegraded":"2024-03-26T06:36:27.562425+0000","last_fullsized":"2024-03-26T06:36:27.562425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:48:47.011884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540090+0000","last_change":"2024-03-26T06:35:27.129880+0000","last_active":"2024-03-26T06:36:27.540090+0000","last_peered":"2024-03-26T06:36:27.540090+0000","last_clean":"2024-03-26T06:36:27.540090+0000","last_became_active":"2024-03-26T06:35:27.129513+0000","last_became_peered":"2024-03-26T06:35:27.129513+0000","last_unstale":"2024-03-26T06:36:27.540090+0000","last_undegraded":"2024-03-26T06:36:27.540090+0000","last_fullsized":"2024-03-26T06:36:27.540090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:10:53.747657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535376+0000","last_change":"2024-03-26T06:35:26.095603+0000","last_active":"2024-03-26T06:36:27.535376+0000","last_peered":"2024-03-26T06:36:27.535376+0000","last_clean":"2024-03-26T06:36:27.535376+0000","last_became_active":"2024-03-26T06:35:26.095309+0000","last_became_peered":"2024-03-26T06:35:26.095309+0000","last_unstale":"2024-03-26T06:36:27.535376+0000","last_undegraded":"2024-03-26T06:36:27.535376+0000","last_fullsized":"2024-03-26T06:36:27.535376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:30:55.373598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.538485+0000","last_change":"2024-03-26T06:35:27.133271+0000","last_active":"2024-03-26T06:36:27.538485+0000","last_peered":"2024-03-26T06:36:27.538485+0000","last_clean":"2024-03-26T06:36:27.538485+0000","last_became_active":"2024-03-26T06:35:27.132553+0000","last_became_peered":"2024-03-26T06:35:27.132553+0000","last_unstale":"2024-03-26T06:36:27.538485+0000","last_undegraded":"2024-03-26T06:36:27.538485+0000","last_fullsized":"2024-03-26T06:36:27.538485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:13:59.021131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:39.275398+0000","last_change":"2024-03-26T06:35:26.094323+0000","last_active":"2024-03-26T06:36:39.275398+0000","last_peered":"2024-03-26T06:36:39.275398+0000","last_clean":"2024-03-26T06:36:39.275398+0000","last_became_active":"2024-03-26T06:35:26.093814+0000","last_became_peered":"2024-03-26T06:35:26.093814+0000","last_unstale":"2024-03-26T06:36:39.275398+0000","last_undegraded":"2024-03-26T06:36:39.275398+0000","last_fullsized":"2024-03-26T06:36:39.275398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:11.450427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534262+0000","last_change":"2024-03-26T06:35:27.127398+0000","last_active":"2024-03-26T06:36:27.534262+0000","last_peered":"2024-03-26T06:36:27.534262+0000","last_clean":"2024-03-26T06:36:27.534262+0000","last_became_active":"2024-03-26T06:35:27.127054+0000","last_became_peered":"2024-03-26T06:35:27.127054+0000","last_unstale":"2024-03-26T06:36:27.534262+0000","last_undegraded":"2024-03-26T06:36:27.534262+0000","last_fullsized":"2024-03-26T06:36:27.534262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:51:04.150954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535969+0000","last_change":"2024-03-26T06:35:26.098531+0000","last_active":"2024-03-26T06:36:27.535969+0000","last_peered":"2024-03-26T06:36:27.535969+0000","last_clean":"2024-03-26T06:36:27.535969+0000","last_became_active":"2024-03-26T06:35:26.097942+0000","last_became_peered":"2024-03-26T06:35:26.097942+0000","last_unstale":"2024-03-26T06:36:27.535969+0000","last_undegraded":"2024-03-26T06:36:27.535969+0000","last_fullsized":"2024-03-26T06:36:27.535969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:44:59.884627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535814+0000","last_change":"2024-03-26T06:35:27.136145+0000","last_active":"2024-03-26T06:36:27.535814+0000","last_peered":"2024-03-26T06:36:27.535814+0000","last_clean":"2024-03-26T06:36:27.535814+0000","last_became_active":"2024-03-26T06:35:27.135560+0000","last_became_peered":"2024-03-26T06:35:27.135560+0000","last_unstale":"2024-03-26T06:36:27.535814+0000","last_undegraded":"2024-03-26T06:36:27.535814+0000","last_fullsized":"2024-03-26T06:36:27.535814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:19:01.991175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536006+0000","last_change":"2024-03-26T06:35:26.095119+0000","last_active":"2024-03-26T06:36:27.536006+0000","last_peered":"2024-03-26T06:36:27.536006+0000","last_clean":"2024-03-26T06:36:27.536006+0000","last_became_active":"2024-03-26T06:35:26.094789+0000","last_became_peered":"2024-03-26T06:35:26.094789+0000","last_unstale":"2024-03-26T06:36:27.536006+0000","last_undegraded":"2024-03-26T06:36:27.536006+0000","last_fullsized":"2024-03-26T06:36:27.536006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:59:07.626207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534210+0000","last_change":"2024-03-26T06:35:27.133722+0000","last_active":"2024-03-26T06:36:27.534210+0000","last_peered":"2024-03-26T06:36:27.534210+0000","last_clean":"2024-03-26T06:36:27.534210+0000","last_became_active":"2024-03-26T06:35:27.132313+0000","last_became_peered":"2024-03-26T06:35:27.132313+0000","last_unstale":"2024-03-26T06:36:27.534210+0000","last_undegraded":"2024-03-26T06:36:27.534210+0000","last_fullsized":"2024-03-26T06:36:27.534210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:58:48.350708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'27","reported_seq":209,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:40.586067+0000","last_change":"2024-03-26T06:36:27.571382+0000","last_active":"2024-03-26T06:36:40.586067+0000","last_peered":"2024-03-26T06:36:40.586067+0000","last_clean":"2024-03-26T06:36:40.586067+0000","last_became_active":"2024-03-26T06:36:27.563425+0000","last_became_peered":"2024-03-26T06:36:27.563425+0000","last_unstale":"2024-03-26T06:36:40.586067+0000","last_undegraded":"2024-03-26T06:36:40.586067+0000","last_fullsized":"2024-03-26T06:36:40.586067+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:21:55.717347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59465,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":119,"num_read_kb":117,"num_write":28,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":59465,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536422+0000","last_change":"2024-03-26T06:35:27.127735+0000","last_active":"2024-03-26T06:36:27.536422+0000","last_peered":"2024-03-26T06:36:27.536422+0000","last_clean":"2024-03-26T06:36:27.536422+0000","last_became_active":"2024-03-26T06:35:27.127434+0000","last_became_peered":"2024-03-26T06:35:27.127434+0000","last_unstale":"2024-03-26T06:36:27.536422+0000","last_undegraded":"2024-03-26T06:36:27.536422+0000","last_fullsized":"2024-03-26T06:36:27.536422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:30:09.745415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534120+0000","last_change":"2024-03-26T06:35:26.095943+0000","last_active":"2024-03-26T06:36:27.534120+0000","last_peered":"2024-03-26T06:36:27.534120+0000","last_clean":"2024-03-26T06:36:27.534120+0000","last_became_active":"2024-03-26T06:35:26.095577+0000","last_became_peered":"2024-03-26T06:35:26.095577+0000","last_unstale":"2024-03-26T06:36:27.534120+0000","last_undegraded":"2024-03-26T06:36:27.534120+0000","last_fullsized":"2024-03-26T06:36:27.534120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:42:34.459051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562209+0000","last_change":"2024-03-26T06:35:27.133199+0000","last_active":"2024-03-26T06:36:27.562209+0000","last_peered":"2024-03-26T06:36:27.562209+0000","last_clean":"2024-03-26T06:36:27.562209+0000","last_became_active":"2024-03-26T06:35:27.132187+0000","last_became_peered":"2024-03-26T06:35:27.132187+0000","last_unstale":"2024-03-26T06:36:27.562209+0000","last_undegraded":"2024-03-26T06:36:27.562209+0000","last_fullsized":"2024-03-26T06:36:27.562209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:16:22.832804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535584+0000","last_change":"2024-03-26T06:35:26.100902+0000","last_active":"2024-03-26T06:36:27.535584+0000","last_peered":"2024-03-26T06:36:27.535584+0000","last_clean":"2024-03-26T06:36:27.535584+0000","last_became_active":"2024-03-26T06:35:26.100509+0000","last_became_peered":"2024-03-26T06:35:26.100509+0000","last_unstale":"2024-03-26T06:36:27.535584+0000","last_undegraded":"2024-03-26T06:36:27.535584+0000","last_fullsized":"2024-03-26T06:36:27.535584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:52:53.459610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535861+0000","last_change":"2024-03-26T06:35:27.129126+0000","last_active":"2024-03-26T06:36:27.535861+0000","last_peered":"2024-03-26T06:36:27.535861+0000","last_clean":"2024-03-26T06:36:27.535861+0000","last_became_active":"2024-03-26T06:35:27.128741+0000","last_became_peered":"2024-03-26T06:35:27.128741+0000","last_unstale":"2024-03-26T06:36:27.535861+0000","last_undegraded":"2024-03-26T06:36:27.535861+0000","last_fullsized":"2024-03-26T06:36:27.535861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:09:15.613369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:40.279330+0000","last_change":"2024-03-26T06:35:26.096239+0000","last_active":"2024-03-26T06:36:40.279330+0000","last_peered":"2024-03-26T06:36:40.279330+0000","last_clean":"2024-03-26T06:36:40.279330+0000","last_became_active":"2024-03-26T06:35:26.095915+0000","last_became_peered":"2024-03-26T06:35:26.095915+0000","last_unstale":"2024-03-26T06:36:40.279330+0000","last_undegraded":"2024-03-26T06:36:40.279330+0000","last_fullsized":"2024-03-26T06:36:40.279330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:50:24.566322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535669+0000","last_change":"2024-03-26T06:35:27.125846+0000","last_active":"2024-03-26T06:36:27.535669+0000","last_peered":"2024-03-26T06:36:27.535669+0000","last_clean":"2024-03-26T06:36:27.535669+0000","last_became_active":"2024-03-26T06:35:27.125515+0000","last_became_peered":"2024-03-26T06:35:27.125515+0000","last_unstale":"2024-03-26T06:36:27.535669+0000","last_undegraded":"2024-03-26T06:36:27.535669+0000","last_fullsized":"2024-03-26T06:36:27.535669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:02:17.358507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535873+0000","last_change":"2024-03-26T06:35:26.102857+0000","last_active":"2024-03-26T06:36:27.535873+0000","last_peered":"2024-03-26T06:36:27.535873+0000","last_clean":"2024-03-26T06:36:27.535873+0000","last_became_active":"2024-03-26T06:35:26.102552+0000","last_became_peered":"2024-03-26T06:35:26.102552+0000","last_unstale":"2024-03-26T06:36:27.535873+0000","last_undegraded":"2024-03-26T06:36:27.535873+0000","last_fullsized":"2024-03-26T06:36:27.535873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:31:30.833012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.537997+0000","last_change":"2024-03-26T06:36:27.537997+0000","last_active":"2024-03-26T06:36:27.537997+0000","last_peered":"2024-03-26T06:36:27.537997+0000","last_clean":"2024-03-26T06:36:27.537997+0000","last_became_active":"2024-03-26T06:36:27.537511+0000","last_became_peered":"2024-03-26T06:36:27.537511+0000","last_unstale":"2024-03-26T06:36:27.537997+0000","last_undegraded":"2024-03-26T06:36:27.537997+0000","last_fullsized":"2024-03-26T06:36:27.537997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:45:11.644668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536538+0000","last_change":"2024-03-26T06:35:26.103876+0000","last_active":"2024-03-26T06:36:27.536538+0000","last_peered":"2024-03-26T06:36:27.536538+0000","last_clean":"2024-03-26T06:36:27.536538+0000","last_became_active":"2024-03-26T06:35:26.102981+0000","last_became_peered":"2024-03-26T06:35:26.102981+0000","last_unstale":"2024-03-26T06:36:27.536538+0000","last_undegraded":"2024-03-26T06:36:27.536538+0000","last_fullsized":"2024-03-26T06:36:27.536538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:24:04.154212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531742+0000","last_change":"2024-03-26T06:35:27.127529+0000","last_active":"2024-03-26T06:36:27.531742+0000","last_peered":"2024-03-26T06:36:27.531742+0000","last_clean":"2024-03-26T06:36:27.531742+0000","last_became_active":"2024-03-26T06:35:27.127208+0000","last_became_peered":"2024-03-26T06:35:27.127208+0000","last_unstale":"2024-03-26T06:36:27.531742+0000","last_undegraded":"2024-03-26T06:36:27.531742+0000","last_fullsized":"2024-03-26T06:36:27.531742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:29:35.313042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.538726+0000","last_change":"2024-03-26T06:35:26.099309+0000","last_active":"2024-03-26T06:36:27.538726+0000","last_peered":"2024-03-26T06:36:27.538726+0000","last_clean":"2024-03-26T06:36:27.538726+0000","last_became_active":"2024-03-26T06:35:26.098948+0000","last_became_peered":"2024-03-26T06:35:26.098948+0000","last_unstale":"2024-03-26T06:36:27.538726+0000","last_undegraded":"2024-03-26T06:36:27.538726+0000","last_fullsized":"2024-03-26T06:36:27.538726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:49:35.210018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.532317+0000","last_change":"2024-03-26T06:35:27.132441+0000","last_active":"2024-03-26T06:36:27.532317+0000","last_peered":"2024-03-26T06:36:27.532317+0000","last_clean":"2024-03-26T06:36:27.532317+0000","last_became_active":"2024-03-26T06:35:27.131846+0000","last_became_peered":"2024-03-26T06:35:27.131846+0000","last_unstale":"2024-03-26T06:36:27.532317+0000","last_undegraded":"2024-03-26T06:36:27.532317+0000","last_fullsized":"2024-03-26T06:36:27.532317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:14.779299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535824+0000","last_change":"2024-03-26T06:35:26.107214+0000","last_active":"2024-03-26T06:36:27.535824+0000","last_peered":"2024-03-26T06:36:27.535824+0000","last_clean":"2024-03-26T06:36:27.535824+0000","last_became_active":"2024-03-26T06:35:26.106935+0000","last_became_peered":"2024-03-26T06:35:26.106935+0000","last_unstale":"2024-03-26T06:36:27.535824+0000","last_undegraded":"2024-03-26T06:36:27.535824+0000","last_fullsized":"2024-03-26T06:36:27.535824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:49:33.101354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.532204+0000","last_change":"2024-03-26T06:35:27.131374+0000","last_active":"2024-03-26T06:36:27.532204+0000","last_peered":"2024-03-26T06:36:27.532204+0000","last_clean":"2024-03-26T06:36:27.532204+0000","last_became_active":"2024-03-26T06:35:27.131030+0000","last_became_peered":"2024-03-26T06:35:27.131030+0000","last_unstale":"2024-03-26T06:36:27.532204+0000","last_undegraded":"2024-03-26T06:36:27.532204+0000","last_fullsized":"2024-03-26T06:36:27.532204+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:33:40.809278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535514+0000","last_change":"2024-03-26T06:35:27.127240+0000","last_active":"2024-03-26T06:36:27.535514+0000","last_peered":"2024-03-26T06:36:27.535514+0000","last_clean":"2024-03-26T06:36:27.535514+0000","last_became_active":"2024-03-26T06:35:27.126923+0000","last_became_peered":"2024-03-26T06:35:27.126923+0000","last_unstale":"2024-03-26T06:36:27.535514+0000","last_undegraded":"2024-03-26T06:36:27.535514+0000","last_fullsized":"2024-03-26T06:36:27.535514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:28:20.422536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":92,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:39.277136+0000","last_change":"2024-03-26T06:35:26.096718+0000","last_active":"2024-03-26T06:36:39.277136+0000","last_peered":"2024-03-26T06:36:39.277136+0000","last_clean":"2024-03-26T06:36:39.277136+0000","last_became_active":"2024-03-26T06:35:26.096407+0000","last_became_peered":"2024-03-26T06:35:26.096407+0000","last_unstale":"2024-03-26T06:36:39.277136+0000","last_undegraded":"2024-03-26T06:36:39.277136+0000","last_fullsized":"2024-03-26T06:36:39.277136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:12:00.799567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540459+0000","last_change":"2024-03-26T06:35:27.138485+0000","last_active":"2024-03-26T06:36:27.540459+0000","last_peered":"2024-03-26T06:36:27.540459+0000","last_clean":"2024-03-26T06:36:27.540459+0000","last_became_active":"2024-03-26T06:35:27.138114+0000","last_became_peered":"2024-03-26T06:35:27.138114+0000","last_unstale":"2024-03-26T06:36:27.540459+0000","last_undegraded":"2024-03-26T06:36:27.540459+0000","last_fullsized":"2024-03-26T06:36:27.540459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:28:25.244683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562961+0000","last_change":"2024-03-26T06:35:26.096111+0000","last_active":"2024-03-26T06:36:27.562961+0000","last_peered":"2024-03-26T06:36:27.562961+0000","last_clean":"2024-03-26T06:36:27.562961+0000","last_became_active":"2024-03-26T06:35:26.095809+0000","last_became_peered":"2024-03-26T06:35:26.095809+0000","last_unstale":"2024-03-26T06:36:27.562961+0000","last_undegraded":"2024-03-26T06:36:27.562961+0000","last_fullsized":"2024-03-26T06:36:27.562961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:04:17.436961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":352,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535806+0000","last_change":"2024-03-26T06:34:13.886034+0000","last_active":"2024-03-26T06:36:27.535806+0000","last_peered":"2024-03-26T06:36:27.535806+0000","last_clean":"2024-03-26T06:36:27.535806+0000","last_became_active":"2024-03-26T06:34:13.885472+0000","last_became_peered":"2024-03-26T06:34:13.885472+0000","last_unstale":"2024-03-26T06:36:27.535806+0000","last_undegraded":"2024-03-26T06:36:27.535806+0000","last_fullsized":"2024-03-26T06:36:27.535806+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:30:52.854577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:30:52.854577+0000","last_clean_scrub_stamp":"2024-03-26T06:30:52.854577+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:43:35.315474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562322+0000","last_change":"2024-03-26T06:35:27.131778+0000","last_active":"2024-03-26T06:36:27.562322+0000","last_peered":"2024-03-26T06:36:27.562322+0000","last_clean":"2024-03-26T06:36:27.562322+0000","last_became_active":"2024-03-26T06:35:27.131433+0000","last_became_peered":"2024-03-26T06:35:27.131433+0000","last_unstale":"2024-03-26T06:36:27.562322+0000","last_undegraded":"2024-03-26T06:36:27.562322+0000","last_fullsized":"2024-03-26T06:36:27.562322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:17:58.614371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535852+0000","last_change":"2024-03-26T06:35:26.106052+0000","last_active":"2024-03-26T06:36:27.535852+0000","last_peered":"2024-03-26T06:36:27.535852+0000","last_clean":"2024-03-26T06:36:27.535852+0000","last_became_active":"2024-03-26T06:35:26.105728+0000","last_became_peered":"2024-03-26T06:35:26.105728+0000","last_unstale":"2024-03-26T06:36:27.535852+0000","last_undegraded":"2024-03-26T06:36:27.535852+0000","last_fullsized":"2024-03-26T06:36:27.535852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:19:12.226026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536562+0000","last_change":"2024-03-26T06:35:26.099487+0000","last_active":"2024-03-26T06:36:27.536562+0000","last_peered":"2024-03-26T06:36:27.536562+0000","last_clean":"2024-03-26T06:36:27.536562+0000","last_became_active":"2024-03-26T06:35:26.099180+0000","last_became_peered":"2024-03-26T06:35:26.099180+0000","last_unstale":"2024-03-26T06:36:27.536562+0000","last_undegraded":"2024-03-26T06:36:27.536562+0000","last_fullsized":"2024-03-26T06:36:27.536562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:15:20.832451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562530+0000","last_change":"2024-03-26T06:35:27.125799+0000","last_active":"2024-03-26T06:36:27.562530+0000","last_peered":"2024-03-26T06:36:27.562530+0000","last_clean":"2024-03-26T06:36:27.562530+0000","last_became_active":"2024-03-26T06:35:27.125323+0000","last_became_peered":"2024-03-26T06:35:27.125323+0000","last_unstale":"2024-03-26T06:36:27.562530+0000","last_undegraded":"2024-03-26T06:36:27.562530+0000","last_fullsized":"2024-03-26T06:36:27.562530+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:07:31.162007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.549340+0000","last_change":"2024-03-26T06:36:27.549340+0000","last_active":"2024-03-26T06:36:27.549340+0000","last_peered":"2024-03-26T06:36:27.549340+0000","last_clean":"2024-03-26T06:36:27.549340+0000","last_became_active":"2024-03-26T06:36:27.541431+0000","last_became_peered":"2024-03-26T06:36:27.541431+0000","last_unstale":"2024-03-26T06:36:27.549340+0000","last_undegraded":"2024-03-26T06:36:27.549340+0000","last_fullsized":"2024-03-26T06:36:27.549340+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:53:43.126462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.541095+0000","last_change":"2024-03-26T06:35:27.128853+0000","last_active":"2024-03-26T06:36:27.541095+0000","last_peered":"2024-03-26T06:36:27.541095+0000","last_clean":"2024-03-26T06:36:27.541095+0000","last_became_active":"2024-03-26T06:35:27.128472+0000","last_became_peered":"2024-03-26T06:35:27.128472+0000","last_unstale":"2024-03-26T06:36:27.541095+0000","last_undegraded":"2024-03-26T06:36:27.541095+0000","last_fullsized":"2024-03-26T06:36:27.541095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:01:01.574623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562404+0000","last_change":"2024-03-26T06:35:26.100523+0000","last_active":"2024-03-26T06:36:27.562404+0000","last_peered":"2024-03-26T06:36:27.562404+0000","last_clean":"2024-03-26T06:36:27.562404+0000","last_became_active":"2024-03-26T06:35:26.099313+0000","last_became_peered":"2024-03-26T06:35:26.099313+0000","last_unstale":"2024-03-26T06:36:27.562404+0000","last_undegraded":"2024-03-26T06:36:27.562404+0000","last_fullsized":"2024-03-26T06:36:27.562404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:43:44.182464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535841+0000","last_change":"2024-03-26T06:35:27.131438+0000","last_active":"2024-03-26T06:36:27.535841+0000","last_peered":"2024-03-26T06:36:27.535841+0000","last_clean":"2024-03-26T06:36:27.535841+0000","last_became_active":"2024-03-26T06:35:27.131112+0000","last_became_peered":"2024-03-26T06:35:27.131112+0000","last_unstale":"2024-03-26T06:36:27.535841+0000","last_undegraded":"2024-03-26T06:36:27.535841+0000","last_fullsized":"2024-03-26T06:36:27.535841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:07.621191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.533821+0000","last_change":"2024-03-26T06:35:26.105039+0000","last_active":"2024-03-26T06:36:27.533821+0000","last_peered":"2024-03-26T06:36:27.533821+0000","last_clean":"2024-03-26T06:36:27.533821+0000","last_became_active":"2024-03-26T06:35:26.104776+0000","last_became_peered":"2024-03-26T06:35:26.104776+0000","last_unstale":"2024-03-26T06:36:27.533821+0000","last_undegraded":"2024-03-26T06:36:27.533821+0000","last_fullsized":"2024-03-26T06:36:27.533821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:20:17.748144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536028+0000","last_change":"2024-03-26T06:35:27.126320+0000","last_active":"2024-03-26T06:36:27.536028+0000","last_peered":"2024-03-26T06:36:27.536028+0000","last_clean":"2024-03-26T06:36:27.536028+0000","last_became_active":"2024-03-26T06:35:27.125996+0000","last_became_peered":"2024-03-26T06:35:27.125996+0000","last_unstale":"2024-03-26T06:36:27.536028+0000","last_undegraded":"2024-03-26T06:36:27.536028+0000","last_fullsized":"2024-03-26T06:36:27.536028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:08:21.386787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540194+0000","last_change":"2024-03-26T06:35:26.103303+0000","last_active":"2024-03-26T06:36:27.540194+0000","last_peered":"2024-03-26T06:36:27.540194+0000","last_clean":"2024-03-26T06:36:27.540194+0000","last_became_active":"2024-03-26T06:35:26.102311+0000","last_became_peered":"2024-03-26T06:35:26.102311+0000","last_unstale":"2024-03-26T06:36:27.540194+0000","last_undegraded":"2024-03-26T06:36:27.540194+0000","last_fullsized":"2024-03-26T06:36:27.540194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:44:13.845386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531613+0000","last_change":"2024-03-26T06:35:27.128845+0000","last_active":"2024-03-26T06:36:27.531613+0000","last_peered":"2024-03-26T06:36:27.531613+0000","last_clean":"2024-03-26T06:36:27.531613+0000","last_became_active":"2024-03-26T06:35:27.128532+0000","last_became_peered":"2024-03-26T06:35:27.128532+0000","last_unstale":"2024-03-26T06:36:27.531613+0000","last_undegraded":"2024-03-26T06:36:27.531613+0000","last_fullsized":"2024-03-26T06:36:27.531613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:35:08.087689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.533849+0000","last_change":"2024-03-26T06:35:26.107589+0000","last_active":"2024-03-26T06:36:27.533849+0000","last_peered":"2024-03-26T06:36:27.533849+0000","last_clean":"2024-03-26T06:36:27.533849+0000","last_became_active":"2024-03-26T06:35:26.107280+0000","last_became_peered":"2024-03-26T06:35:26.107280+0000","last_unstale":"2024-03-26T06:36:27.533849+0000","last_undegraded":"2024-03-26T06:36:27.533849+0000","last_fullsized":"2024-03-26T06:36:27.533849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:57:27.621484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540508+0000","last_change":"2024-03-26T06:35:27.131238+0000","last_active":"2024-03-26T06:36:27.540508+0000","last_peered":"2024-03-26T06:36:27.540508+0000","last_clean":"2024-03-26T06:36:27.540508+0000","last_became_active":"2024-03-26T06:35:27.130902+0000","last_became_peered":"2024-03-26T06:35:27.130902+0000","last_unstale":"2024-03-26T06:36:27.540508+0000","last_undegraded":"2024-03-26T06:36:27.540508+0000","last_fullsized":"2024-03-26T06:36:27.540508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:02:47.180199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.567118+0000","last_change":"2024-03-26T06:36:27.567118+0000","last_active":"2024-03-26T06:36:27.567118+0000","last_peered":"2024-03-26T06:36:27.567118+0000","last_clean":"2024-03-26T06:36:27.567118+0000","last_became_active":"2024-03-26T06:36:27.563836+0000","last_became_peered":"2024-03-26T06:36:27.563836+0000","last_unstale":"2024-03-26T06:36:27.567118+0000","last_undegraded":"2024-03-26T06:36:27.567118+0000","last_fullsized":"2024-03-26T06:36:27.567118+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:21:00.417615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535830+0000","last_change":"2024-03-26T06:35:27.133273+0000","last_active":"2024-03-26T06:36:27.535830+0000","last_peered":"2024-03-26T06:36:27.535830+0000","last_clean":"2024-03-26T06:36:27.535830+0000","last_became_active":"2024-03-26T06:35:27.132955+0000","last_became_peered":"2024-03-26T06:35:27.132955+0000","last_unstale":"2024-03-26T06:36:27.535830+0000","last_undegraded":"2024-03-26T06:36:27.535830+0000","last_fullsized":"2024-03-26T06:36:27.535830+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:26:01.542473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534470+0000","last_change":"2024-03-26T06:35:26.103065+0000","last_active":"2024-03-26T06:36:27.534470+0000","last_peered":"2024-03-26T06:36:27.534470+0000","last_clean":"2024-03-26T06:36:27.534470+0000","last_became_active":"2024-03-26T06:35:26.102761+0000","last_became_peered":"2024-03-26T06:35:26.102761+0000","last_unstale":"2024-03-26T06:36:27.534470+0000","last_undegraded":"2024-03-26T06:36:27.534470+0000","last_fullsized":"2024-03-26T06:36:27.534470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:09:27.650223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536071+0000","last_change":"2024-03-26T06:35:27.137820+0000","last_active":"2024-03-26T06:36:27.536071+0000","last_peered":"2024-03-26T06:36:27.536071+0000","last_clean":"2024-03-26T06:36:27.536071+0000","last_became_active":"2024-03-26T06:35:27.137375+0000","last_became_peered":"2024-03-26T06:35:27.137375+0000","last_unstale":"2024-03-26T06:36:27.536071+0000","last_undegraded":"2024-03-26T06:36:27.536071+0000","last_fullsized":"2024-03-26T06:36:27.536071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:43:13.908662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535800+0000","last_change":"2024-03-26T06:35:26.094295+0000","last_active":"2024-03-26T06:36:27.535800+0000","last_peered":"2024-03-26T06:36:27.535800+0000","last_clean":"2024-03-26T06:36:27.535800+0000","last_became_active":"2024-03-26T06:35:26.093950+0000","last_became_peered":"2024-03-26T06:35:26.093950+0000","last_unstale":"2024-03-26T06:36:27.535800+0000","last_undegraded":"2024-03-26T06:36:27.535800+0000","last_fullsized":"2024-03-26T06:36:27.535800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:45:23.700237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535932+0000","last_change":"2024-03-26T06:35:27.131253+0000","last_active":"2024-03-26T06:36:27.535932+0000","last_peered":"2024-03-26T06:36:27.535932+0000","last_clean":"2024-03-26T06:36:27.535932+0000","last_became_active":"2024-03-26T06:35:27.130929+0000","last_became_peered":"2024-03-26T06:35:27.130929+0000","last_unstale":"2024-03-26T06:36:27.535932+0000","last_undegraded":"2024-03-26T06:36:27.535932+0000","last_fullsized":"2024-03-26T06:36:27.535932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:45:55.744139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562582+0000","last_change":"2024-03-26T06:35:26.104272+0000","last_active":"2024-03-26T06:36:27.562582+0000","last_peered":"2024-03-26T06:36:27.562582+0000","last_clean":"2024-03-26T06:36:27.562582+0000","last_became_active":"2024-03-26T06:35:26.103957+0000","last_became_peered":"2024-03-26T06:35:26.103957+0000","last_unstale":"2024-03-26T06:36:27.562582+0000","last_undegraded":"2024-03-26T06:36:27.562582+0000","last_fullsized":"2024-03-26T06:36:27.562582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:09:29.037044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536075+0000","last_change":"2024-03-26T06:35:27.128221+0000","last_active":"2024-03-26T06:36:27.536075+0000","last_peered":"2024-03-26T06:36:27.536075+0000","last_clean":"2024-03-26T06:36:27.536075+0000","last_became_active":"2024-03-26T06:35:27.126609+0000","last_became_peered":"2024-03-26T06:35:27.126609+0000","last_unstale":"2024-03-26T06:36:27.536075+0000","last_undegraded":"2024-03-26T06:36:27.536075+0000","last_fullsized":"2024-03-26T06:36:27.536075+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:59:56.314853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.539868+0000","last_change":"2024-03-26T06:36:27.539868+0000","last_active":"2024-03-26T06:36:27.539868+0000","last_peered":"2024-03-26T06:36:27.539868+0000","last_clean":"2024-03-26T06:36:27.539868+0000","last_became_active":"2024-03-26T06:36:27.539468+0000","last_became_peered":"2024-03-26T06:36:27.539468+0000","last_unstale":"2024-03-26T06:36:27.539868+0000","last_undegraded":"2024-03-26T06:36:27.539868+0000","last_fullsized":"2024-03-26T06:36:27.539868+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:56:33.174588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534265+0000","last_change":"2024-03-26T06:35:27.127975+0000","last_active":"2024-03-26T06:36:27.534265+0000","last_peered":"2024-03-26T06:36:27.534265+0000","last_clean":"2024-03-26T06:36:27.534265+0000","last_became_active":"2024-03-26T06:35:27.127635+0000","last_became_peered":"2024-03-26T06:35:27.127635+0000","last_unstale":"2024-03-26T06:36:27.534265+0000","last_undegraded":"2024-03-26T06:36:27.534265+0000","last_fullsized":"2024-03-26T06:36:27.534265+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:48.587117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.533820+0000","last_change":"2024-03-26T06:35:27.133538+0000","last_active":"2024-03-26T06:36:27.533820+0000","last_peered":"2024-03-26T06:36:27.533820+0000","last_clean":"2024-03-26T06:36:27.533820+0000","last_became_active":"2024-03-26T06:35:27.133104+0000","last_became_peered":"2024-03-26T06:35:27.133104+0000","last_unstale":"2024-03-26T06:36:27.533820+0000","last_undegraded":"2024-03-26T06:36:27.533820+0000","last_fullsized":"2024-03-26T06:36:27.533820+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:20:10.417567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535554+0000","last_change":"2024-03-26T06:35:27.129503+0000","last_active":"2024-03-26T06:36:27.535554+0000","last_peered":"2024-03-26T06:36:27.535554+0000","last_clean":"2024-03-26T06:36:27.535554+0000","last_became_active":"2024-03-26T06:35:27.128930+0000","last_became_peered":"2024-03-26T06:35:27.128930+0000","last_unstale":"2024-03-26T06:36:27.535554+0000","last_undegraded":"2024-03-26T06:36:27.535554+0000","last_fullsized":"2024-03-26T06:36:27.535554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:23:37.476593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562432+0000","last_change":"2024-03-26T06:35:27.137385+0000","last_active":"2024-03-26T06:36:27.562432+0000","last_peered":"2024-03-26T06:36:27.562432+0000","last_clean":"2024-03-26T06:36:27.562432+0000","last_became_active":"2024-03-26T06:35:27.136969+0000","last_became_peered":"2024-03-26T06:35:27.136969+0000","last_unstale":"2024-03-26T06:36:27.562432+0000","last_undegraded":"2024-03-26T06:36:27.562432+0000","last_fullsized":"2024-03-26T06:36:27.562432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:40:16.722381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531790+0000","last_change":"2024-03-26T06:35:27.125287+0000","last_active":"2024-03-26T06:36:27.531790+0000","last_peered":"2024-03-26T06:36:27.531790+0000","last_clean":"2024-03-26T06:36:27.531790+0000","last_became_active":"2024-03-26T06:35:27.124274+0000","last_became_peered":"2024-03-26T06:35:27.124274+0000","last_unstale":"2024-03-26T06:36:27.531790+0000","last_undegraded":"2024-03-26T06:36:27.531790+0000","last_fullsized":"2024-03-26T06:36:27.531790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:33:43.461949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534081+0000","last_change":"2024-03-26T06:35:27.124805+0000","last_active":"2024-03-26T06:36:27.534081+0000","last_peered":"2024-03-26T06:36:27.534081+0000","last_clean":"2024-03-26T06:36:27.534081+0000","last_became_active":"2024-03-26T06:35:27.123947+0000","last_became_peered":"2024-03-26T06:35:27.123947+0000","last_unstale":"2024-03-26T06:36:27.534081+0000","last_undegraded":"2024-03-26T06:36:27.534081+0000","last_fullsized":"2024-03-26T06:36:27.534081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:47.273083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540556+0000","last_change":"2024-03-26T06:35:27.141167+0000","last_active":"2024-03-26T06:36:27.540556+0000","last_peered":"2024-03-26T06:36:27.540556+0000","last_clean":"2024-03-26T06:36:27.540556+0000","last_became_active":"2024-03-26T06:35:27.140781+0000","last_became_peered":"2024-03-26T06:35:27.140781+0000","last_unstale":"2024-03-26T06:36:27.540556+0000","last_undegraded":"2024-03-26T06:36:27.540556+0000","last_fullsized":"2024-03-26T06:36:27.540556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:42:36.705671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535401+0000","last_change":"2024-03-26T06:35:27.132336+0000","last_active":"2024-03-26T06:36:27.535401+0000","last_peered":"2024-03-26T06:36:27.535401+0000","last_clean":"2024-03-26T06:36:27.535401+0000","last_became_active":"2024-03-26T06:35:27.131944+0000","last_became_peered":"2024-03-26T06:35:27.131944+0000","last_unstale":"2024-03-26T06:36:27.535401+0000","last_undegraded":"2024-03-26T06:36:27.535401+0000","last_fullsized":"2024-03-26T06:36:27.535401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:51:57.916873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540170+0000","last_change":"2024-03-26T06:35:27.141297+0000","last_active":"2024-03-26T06:36:27.540170+0000","last_peered":"2024-03-26T06:36:27.540170+0000","last_clean":"2024-03-26T06:36:27.540170+0000","last_became_active":"2024-03-26T06:35:27.140932+0000","last_became_peered":"2024-03-26T06:35:27.140932+0000","last_unstale":"2024-03-26T06:36:27.540170+0000","last_undegraded":"2024-03-26T06:36:27.540170+0000","last_fullsized":"2024-03-26T06:36:27.540170+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:17.942885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534003+0000","last_change":"2024-03-26T06:35:27.135381+0000","last_active":"2024-03-26T06:36:27.534003+0000","last_peered":"2024-03-26T06:36:27.534003+0000","last_clean":"2024-03-26T06:36:27.534003+0000","last_became_active":"2024-03-26T06:35:27.135053+0000","last_became_peered":"2024-03-26T06:35:27.135053+0000","last_unstale":"2024-03-26T06:36:27.534003+0000","last_undegraded":"2024-03-26T06:36:27.534003+0000","last_fullsized":"2024-03-26T06:36:27.534003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:40:21.960182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562542+0000","last_change":"2024-03-26T06:35:27.130780+0000","last_active":"2024-03-26T06:36:27.562542+0000","last_peered":"2024-03-26T06:36:27.562542+0000","last_clean":"2024-03-26T06:36:27.562542+0000","last_became_active":"2024-03-26T06:35:27.130171+0000","last_became_peered":"2024-03-26T06:35:27.130171+0000","last_unstale":"2024-03-26T06:36:27.562542+0000","last_undegraded":"2024-03-26T06:36:27.562542+0000","last_fullsized":"2024-03-26T06:36:27.562542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:53:18.053617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531641+0000","last_change":"2024-03-26T06:35:27.129677+0000","last_active":"2024-03-26T06:36:27.531641+0000","last_peered":"2024-03-26T06:36:27.531641+0000","last_clean":"2024-03-26T06:36:27.531641+0000","last_became_active":"2024-03-26T06:35:27.129275+0000","last_became_peered":"2024-03-26T06:35:27.129275+0000","last_unstale":"2024-03-26T06:36:27.531641+0000","last_undegraded":"2024-03-26T06:36:27.531641+0000","last_fullsized":"2024-03-26T06:36:27.531641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:18.851198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63466,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":414,"num_read_kb":274,"num_write":155,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":60047,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":190398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351005,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":459933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612641,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1654784,"data_stored":1110688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776166,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":459587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037803,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":974848,"data_stored":459507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299440,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32712,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712440,"statfs":{"total":95995035648,"available":95961538560,"internally_reserved":0,"allocated":1564672,"data_stored":1050855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462966,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":458976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":518234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":460247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411402,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40460,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704692,"statfs":{"total":95995035648,"available":95953604608,"internally_reserved":0,"allocated":978944,"data_stored":460119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":459287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888128,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":519269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724602,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1581056,"data_stored":1050730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T06:36:44.515 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T06:36:44.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:44 smithi121 ceph-mon[36349]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s; 4.9 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-26T06:36:44.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:44 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:36:44.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:44 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:36:44.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:44 smithi121 ceph-mon[36349]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:44.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:44 smithi170 ceph-mon[36912]: pgmap v326: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s; 4.9 KiB/s, 0 keys/s, 0 objects/s recovering 2024-03-26T06:36:44.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:44 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:36:44.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:44 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:36:44.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:44 smithi170 ceph-mon[36912]: from='mgr.14152 172.21.15.38:0/762033383' entity='mgr.x' 2024-03-26T06:36:44.759 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:45.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:45 smithi038 ceph-mon[32265]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:36:45.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:45 smithi121 ceph-mon[36349]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:36:45.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:45 smithi170 ceph-mon[36912]: from='client.14958 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:36:46.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:46 smithi038 ceph-mon[32265]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:46.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.478+0000 7f39e42e1700 1 -- 172.21.15.38:0/227204074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc101680 msgr2=0x7f39dc10db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:46.501 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.478+0000 7f39e42e1700 1 --2- 172.21.15.38:0/227204074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc101680 0x7f39dc10db60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f39d4009990 tx=0x7f39d4009ca0 comp rx=0 tx=0).stop 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.478+0000 7f39e42e1700 1 -- 172.21.15.38:0/227204074 shutdown_connections 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.478+0000 7f39e42e1700 1 --2- 172.21.15.38:0/227204074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc101680 0x7f39dc10db60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.478+0000 7f39e42e1700 1 --2- 172.21.15.38:0/227204074 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39dc106d20 0x7f39dc107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.478+0000 7f39e42e1700 1 --2- 172.21.15.38:0/227204074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39dc100ce0 0x7f39dc101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.478+0000 7f39e42e1700 1 -- 172.21.15.38:0/227204074 >> 172.21.15.38:0/227204074 conn(0x7f39dc077c10 msgr2=0x7f39dc078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.478+0000 7f39e42e1700 1 -- 172.21.15.38:0/227204074 shutdown_connections 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.478+0000 7f39e42e1700 1 -- 172.21.15.38:0/227204074 wait complete. 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.479+0000 7f39e42e1700 1 Processor -- start 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.479+0000 7f39e42e1700 1 -- start start 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.479+0000 7f39e42e1700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39dc100ce0 0x7f39dc19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.479+0000 7f39e42e1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc101680 0x7f39dc19ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.479+0000 7f39e42e1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39dc106d20 0x7f39dc19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:46.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.479+0000 7f39e42e1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc110fe0 con 0x7f39dc101680 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.479+0000 7f39e42e1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc110e60 con 0x7f39dc106d20 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.479+0000 7f39e42e1700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39dc111160 con 0x7f39dc100ce0 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.479+0000 7f39e187c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc101680 0x7f39dc19ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.479+0000 7f39e187c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc101680 0x7f39dc19ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43100/0 (socket says 172.21.15.38:43100) 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.479+0000 7f39e187c700 1 -- 172.21.15.38:0/3781219558 learned_addr learned my addr 172.21.15.38:0/3781219558 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39e207d700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39dc100ce0 0x7f39dc19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39e287e700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39dc106d20 0x7f39dc19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39e187c700 1 -- 172.21.15.38:0/3781219558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39dc100ce0 msgr2=0x7f39dc19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39e187c700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39dc100ce0 0x7f39dc19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39e187c700 1 -- 172.21.15.38:0/3781219558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39dc106d20 msgr2=0x7f39dc19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39e187c700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39dc106d20 0x7f39dc19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39e187c700 1 -- 172.21.15.38:0/3781219558 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39dc19f6e0 con 0x7f39dc101680 2024-03-26T06:36:46.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39e187c700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc101680 0x7f39dc19ac10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f39d000c9b0 tx=0x7f39d000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39db7fe700 1 -- 172.21.15.38:0/3781219558 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0007b00 con 0x7f39dc101680 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39e42e1700 1 -- 172.21.15.38:0/3781219558 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39dc19f9d0 con 0x7f39dc101680 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39db7fe700 1 -- 172.21.15.38:0/3781219558 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39d0007ca0 con 0x7f39dc101680 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39db7fe700 1 -- 172.21.15.38:0/3781219558 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0018730 con 0x7f39dc101680 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.480+0000 7f39e42e1700 1 -- 172.21.15.38:0/3781219558 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39dc1a7250 con 0x7f39dc101680 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.481+0000 7f39e42e1700 1 -- 172.21.15.38:0/3781219558 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39dc068aa0 con 0x7f39dc101680 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.482+0000 7f39db7fe700 1 -- 172.21.15.38:0/3781219558 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f39d00188d0 con 0x7f39dc101680 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.483+0000 7f39db7fe700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f39c40a2920 0x7f39c40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.483+0000 7f39e207d700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f39c40a2920 0x7f39c40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.483+0000 7f39db7fe700 1 -- 172.21.15.38:0/3781219558 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f39d00cee80 con 0x7f39dc101680 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.483+0000 7f39e207d700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f39c40a2920 0x7f39c40a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00a400 tx=0x7f39cc00d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:46.504 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.487+0000 7f39db7fe700 1 -- 172.21.15.38:0/3781219558 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f39d0014030 con 0x7f39dc101680 2024-03-26T06:36:46.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:46 smithi121 ceph-mon[36349]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:46.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:46 smithi170 ceph-mon[36912]: pgmap v327: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:46.690 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.687+0000 7f39e42e1700 1 -- 172.21.15.38:0/3781219558 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39dc100360 con 0x7f39c40a2920 2024-03-26T06:36:46.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.695+0000 7f39db7fe700 1 -- 172.21.15.38:0/3781219558 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258721 (secure 0 0 0) 0x7f39dc100360 con 0x7f39c40a2920 2024-03-26T06:36:46.701 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.697+0000 7f39e42e1700 1 -- 172.21.15.38:0/3781219558 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f39c40a2920 msgr2=0x7f39c40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.697+0000 7f39e42e1700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f39c40a2920 0x7f39c40a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f39cc00a400 tx=0x7f39cc00d040 comp rx=0 tx=0).stop 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.697+0000 7f39e42e1700 1 -- 172.21.15.38:0/3781219558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc101680 msgr2=0x7f39dc19ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.697+0000 7f39e42e1700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc101680 0x7f39dc19ac10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f39d000c9b0 tx=0x7f39d000ccc0 comp rx=0 tx=0).stop 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.697+0000 7f39e42e1700 1 -- 172.21.15.38:0/3781219558 shutdown_connections 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.697+0000 7f39e42e1700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f39c40a2920 0x7f39c40a4de0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.697+0000 7f39e42e1700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39dc106d20 0x7f39dc19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.697+0000 7f39e42e1700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f39dc100ce0 0x7f39dc19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.697+0000 7f39e42e1700 1 --2- 172.21.15.38:0/3781219558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f39dc101680 0x7f39dc19ac10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.697+0000 7f39e42e1700 1 -- 172.21.15.38:0/3781219558 >> 172.21.15.38:0/3781219558 conn(0x7f39dc077c10 msgr2=0x7f39dc0ffc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.697+0000 7f39e42e1700 1 -- 172.21.15.38:0/3781219558 shutdown_connections 2024-03-26T06:36:46.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:46.698+0000 7f39e42e1700 1 -- 172.21.15.38:0/3781219558 wait complete. 2024-03-26T06:36:46.705 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2024-03-26T06:36:47.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:47 smithi038 ceph-mon[32265]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:47.481 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:47 smithi038 ceph-mon[32265]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:36:47.510 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-03-26T06:36:46.269574+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654068,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":575,"num_read_kb":408,"num_write":338,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1240783,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":370912,"kb_used_data":13356,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124570912,"statfs":{"total":1151940427776,"available":1151560613888,"internally_reserved":0,"allocated":13676544,"data_stored":7467432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":45,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.019600"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531887+0000","last_change":"2024-03-26T06:35:27.137165+0000","last_active":"2024-03-26T06:36:27.531887+0000","last_peered":"2024-03-26T06:36:27.531887+0000","last_clean":"2024-03-26T06:36:27.531887+0000","last_became_active":"2024-03-26T06:35:27.136869+0000","last_became_peered":"2024-03-26T06:35:27.136869+0000","last_unstale":"2024-03-26T06:36:27.531887+0000","last_undegraded":"2024-03-26T06:36:27.531887+0000","last_fullsized":"2024-03-26T06:36:27.531887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:43:36.671967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535676+0000","last_change":"2024-03-26T06:35:27.130619+0000","last_active":"2024-03-26T06:36:27.535676+0000","last_peered":"2024-03-26T06:36:27.535676+0000","last_clean":"2024-03-26T06:36:27.535676+0000","last_became_active":"2024-03-26T06:35:27.128846+0000","last_became_peered":"2024-03-26T06:35:27.128846+0000","last_unstale":"2024-03-26T06:36:27.535676+0000","last_undegraded":"2024-03-26T06:36:27.535676+0000","last_fullsized":"2024-03-26T06:36:27.535676+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:14:15.627268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535568+0000","last_change":"2024-03-26T06:35:27.138214+0000","last_active":"2024-03-26T06:36:27.535568+0000","last_peered":"2024-03-26T06:36:27.535568+0000","last_clean":"2024-03-26T06:36:27.535568+0000","last_became_active":"2024-03-26T06:35:27.137846+0000","last_became_peered":"2024-03-26T06:35:27.137846+0000","last_unstale":"2024-03-26T06:36:27.535568+0000","last_undegraded":"2024-03-26T06:36:27.535568+0000","last_fullsized":"2024-03-26T06:36:27.535568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:48:46.441889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562934+0000","last_change":"2024-03-26T06:35:27.129582+0000","last_active":"2024-03-26T06:36:27.562934+0000","last_peered":"2024-03-26T06:36:27.562934+0000","last_clean":"2024-03-26T06:36:27.562934+0000","last_became_active":"2024-03-26T06:35:27.129245+0000","last_became_peered":"2024-03-26T06:35:27.129245+0000","last_unstale":"2024-03-26T06:36:27.562934+0000","last_undegraded":"2024-03-26T06:36:27.562934+0000","last_fullsized":"2024-03-26T06:36:27.562934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:09:14.548067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536655+0000","last_change":"2024-03-26T06:35:27.131993+0000","last_active":"2024-03-26T06:36:27.536655+0000","last_peered":"2024-03-26T06:36:27.536655+0000","last_clean":"2024-03-26T06:36:27.536655+0000","last_became_active":"2024-03-26T06:35:27.131650+0000","last_became_peered":"2024-03-26T06:35:27.131650+0000","last_unstale":"2024-03-26T06:36:27.536655+0000","last_undegraded":"2024-03-26T06:36:27.536655+0000","last_fullsized":"2024-03-26T06:36:27.536655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:38:41.407314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.532074+0000","last_change":"2024-03-26T06:35:27.130325+0000","last_active":"2024-03-26T06:36:27.532074+0000","last_peered":"2024-03-26T06:36:27.532074+0000","last_clean":"2024-03-26T06:36:27.532074+0000","last_became_active":"2024-03-26T06:35:27.129658+0000","last_became_peered":"2024-03-26T06:35:27.129658+0000","last_unstale":"2024-03-26T06:36:27.532074+0000","last_undegraded":"2024-03-26T06:36:27.532074+0000","last_fullsized":"2024-03-26T06:36:27.532074+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:25:18.580839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535725+0000","last_change":"2024-03-26T06:35:27.130870+0000","last_active":"2024-03-26T06:36:27.535725+0000","last_peered":"2024-03-26T06:36:27.535725+0000","last_clean":"2024-03-26T06:36:27.535725+0000","last_became_active":"2024-03-26T06:35:27.130521+0000","last_became_peered":"2024-03-26T06:35:27.130521+0000","last_unstale":"2024-03-26T06:36:27.535725+0000","last_undegraded":"2024-03-26T06:36:27.535725+0000","last_fullsized":"2024-03-26T06:36:27.535725+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:58:40.640708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540027+0000","last_change":"2024-03-26T06:35:27.136258+0000","last_active":"2024-03-26T06:36:27.540027+0000","last_peered":"2024-03-26T06:36:27.540027+0000","last_clean":"2024-03-26T06:36:27.540027+0000","last_became_active":"2024-03-26T06:35:27.135746+0000","last_became_peered":"2024-03-26T06:35:27.135746+0000","last_unstale":"2024-03-26T06:36:27.540027+0000","last_undegraded":"2024-03-26T06:36:27.540027+0000","last_fullsized":"2024-03-26T06:36:27.540027+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:42:24.270717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534322+0000","last_change":"2024-03-26T06:35:27.130533+0000","last_active":"2024-03-26T06:36:27.534322+0000","last_peered":"2024-03-26T06:36:27.534322+0000","last_clean":"2024-03-26T06:36:27.534322+0000","last_became_active":"2024-03-26T06:35:27.130251+0000","last_became_peered":"2024-03-26T06:35:27.130251+0000","last_unstale":"2024-03-26T06:36:27.534322+0000","last_undegraded":"2024-03-26T06:36:27.534322+0000","last_fullsized":"2024-03-26T06:36:27.534322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T18:13:40.209389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.539256+0000","last_change":"2024-03-26T06:36:27.539256+0000","last_active":"2024-03-26T06:36:27.539256+0000","last_peered":"2024-03-26T06:36:27.539256+0000","last_clean":"2024-03-26T06:36:27.539256+0000","last_became_active":"2024-03-26T06:36:27.538934+0000","last_became_peered":"2024-03-26T06:36:27.538934+0000","last_unstale":"2024-03-26T06:36:27.539256+0000","last_undegraded":"2024-03-26T06:36:27.539256+0000","last_fullsized":"2024-03-26T06:36:27.539256+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:29:04.582141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536010+0000","last_change":"2024-03-26T06:35:27.137285+0000","last_active":"2024-03-26T06:36:27.536010+0000","last_peered":"2024-03-26T06:36:27.536010+0000","last_clean":"2024-03-26T06:36:27.536010+0000","last_became_active":"2024-03-26T06:35:27.136930+0000","last_became_peered":"2024-03-26T06:35:27.136930+0000","last_unstale":"2024-03-26T06:36:27.536010+0000","last_undegraded":"2024-03-26T06:36:27.536010+0000","last_fullsized":"2024-03-26T06:36:27.536010+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:36:42.036591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562901+0000","last_change":"2024-03-26T06:35:27.133279+0000","last_active":"2024-03-26T06:36:27.562901+0000","last_peered":"2024-03-26T06:36:27.562901+0000","last_clean":"2024-03-26T06:36:27.562901+0000","last_became_active":"2024-03-26T06:35:27.132860+0000","last_became_peered":"2024-03-26T06:35:27.132860+0000","last_unstale":"2024-03-26T06:36:27.562901+0000","last_undegraded":"2024-03-26T06:36:27.562901+0000","last_fullsized":"2024-03-26T06:36:27.562901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:36:34.455411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562469+0000","last_change":"2024-03-26T06:35:27.124143+0000","last_active":"2024-03-26T06:36:27.562469+0000","last_peered":"2024-03-26T06:36:27.562469+0000","last_clean":"2024-03-26T06:36:27.562469+0000","last_became_active":"2024-03-26T06:35:27.123774+0000","last_became_peered":"2024-03-26T06:35:27.123774+0000","last_unstale":"2024-03-26T06:36:27.562469+0000","last_undegraded":"2024-03-26T06:36:27.562469+0000","last_fullsized":"2024-03-26T06:36:27.562469+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:05:14.135531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535360+0000","last_change":"2024-03-26T06:35:27.127523+0000","last_active":"2024-03-26T06:36:27.535360+0000","last_peered":"2024-03-26T06:36:27.535360+0000","last_clean":"2024-03-26T06:36:27.535360+0000","last_became_active":"2024-03-26T06:35:27.127141+0000","last_became_peered":"2024-03-26T06:35:27.127141+0000","last_unstale":"2024-03-26T06:36:27.535360+0000","last_undegraded":"2024-03-26T06:36:27.535360+0000","last_fullsized":"2024-03-26T06:36:27.535360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:21:44.524721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.563452+0000","last_change":"2024-03-26T06:36:27.563452+0000","last_active":"2024-03-26T06:36:27.563452+0000","last_peered":"2024-03-26T06:36:27.563452+0000","last_clean":"2024-03-26T06:36:27.563452+0000","last_became_active":"2024-03-26T06:36:27.563070+0000","last_became_peered":"2024-03-26T06:36:27.563070+0000","last_unstale":"2024-03-26T06:36:27.563452+0000","last_undegraded":"2024-03-26T06:36:27.563452+0000","last_fullsized":"2024-03-26T06:36:27.563452+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:14:41.816751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.533805+0000","last_change":"2024-03-26T06:35:27.132021+0000","last_active":"2024-03-26T06:36:27.533805+0000","last_peered":"2024-03-26T06:36:27.533805+0000","last_clean":"2024-03-26T06:36:27.533805+0000","last_became_active":"2024-03-26T06:35:27.131722+0000","last_became_peered":"2024-03-26T06:35:27.131722+0000","last_unstale":"2024-03-26T06:36:27.533805+0000","last_undegraded":"2024-03-26T06:36:27.533805+0000","last_fullsized":"2024-03-26T06:36:27.533805+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:52:23.253250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531749+0000","last_change":"2024-03-26T06:35:27.137225+0000","last_active":"2024-03-26T06:36:27.531749+0000","last_peered":"2024-03-26T06:36:27.531749+0000","last_clean":"2024-03-26T06:36:27.531749+0000","last_became_active":"2024-03-26T06:35:27.136935+0000","last_became_peered":"2024-03-26T06:35:27.136935+0000","last_unstale":"2024-03-26T06:36:27.531749+0000","last_undegraded":"2024-03-26T06:36:27.531749+0000","last_fullsized":"2024-03-26T06:36:27.531749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:14:02.540482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.538699+0000","last_change":"2024-03-26T06:35:27.136068+0000","last_active":"2024-03-26T06:36:27.538699+0000","last_peered":"2024-03-26T06:36:27.538699+0000","last_clean":"2024-03-26T06:36:27.538699+0000","last_became_active":"2024-03-26T06:35:27.135586+0000","last_became_peered":"2024-03-26T06:35:27.135586+0000","last_unstale":"2024-03-26T06:36:27.538699+0000","last_undegraded":"2024-03-26T06:36:27.538699+0000","last_fullsized":"2024-03-26T06:36:27.538699+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:21:51.729675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.563455+0000","last_change":"2024-03-26T06:36:27.563455+0000","last_active":"2024-03-26T06:36:27.563455+0000","last_peered":"2024-03-26T06:36:27.563455+0000","last_clean":"2024-03-26T06:36:27.563455+0000","last_became_active":"2024-03-26T06:36:27.563115+0000","last_became_peered":"2024-03-26T06:36:27.563115+0000","last_unstale":"2024-03-26T06:36:27.563455+0000","last_undegraded":"2024-03-26T06:36:27.563455+0000","last_fullsized":"2024-03-26T06:36:27.563455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:30:17.781959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,6],"acting":[2,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":45,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.546513+0000","last_change":"2024-03-26T06:36:27.546513+0000","last_active":"2024-03-26T06:36:27.546513+0000","last_peered":"2024-03-26T06:36:27.546513+0000","last_clean":"2024-03-26T06:36:27.546513+0000","last_became_active":"2024-03-26T06:36:27.540060+0000","last_became_peered":"2024-03-26T06:36:27.540060+0000","last_unstale":"2024-03-26T06:36:27.546513+0000","last_undegraded":"2024-03-26T06:36:27.546513+0000","last_fullsized":"2024-03-26T06:36:27.546513+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:22:58.631595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,8],"acting":[9,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540849+0000","last_change":"2024-03-26T06:35:27.141510+0000","last_active":"2024-03-26T06:36:27.540849+0000","last_peered":"2024-03-26T06:36:27.540849+0000","last_clean":"2024-03-26T06:36:27.540849+0000","last_became_active":"2024-03-26T06:35:27.141182+0000","last_became_peered":"2024-03-26T06:35:27.141182+0000","last_unstale":"2024-03-26T06:36:27.540849+0000","last_undegraded":"2024-03-26T06:36:27.540849+0000","last_fullsized":"2024-03-26T06:36:27.540849+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:40:55.993939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.541017+0000","last_change":"2024-03-26T06:35:26.095502+0000","last_active":"2024-03-26T06:36:27.541017+0000","last_peered":"2024-03-26T06:36:27.541017+0000","last_clean":"2024-03-26T06:36:27.541017+0000","last_became_active":"2024-03-26T06:35:26.095168+0000","last_became_peered":"2024-03-26T06:35:26.095168+0000","last_unstale":"2024-03-26T06:36:27.541017+0000","last_undegraded":"2024-03-26T06:36:27.541017+0000","last_fullsized":"2024-03-26T06:36:27.541017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:01:44.307274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535881+0000","last_change":"2024-03-26T06:35:27.137179+0000","last_active":"2024-03-26T06:36:27.535881+0000","last_peered":"2024-03-26T06:36:27.535881+0000","last_clean":"2024-03-26T06:36:27.535881+0000","last_became_active":"2024-03-26T06:35:27.136660+0000","last_became_peered":"2024-03-26T06:35:27.136660+0000","last_unstale":"2024-03-26T06:36:27.535881+0000","last_undegraded":"2024-03-26T06:36:27.535881+0000","last_fullsized":"2024-03-26T06:36:27.535881+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:16:16.042134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536655+0000","last_change":"2024-03-26T06:35:26.105395+0000","last_active":"2024-03-26T06:36:27.536655+0000","last_peered":"2024-03-26T06:36:27.536655+0000","last_clean":"2024-03-26T06:36:27.536655+0000","last_became_active":"2024-03-26T06:35:26.104879+0000","last_became_peered":"2024-03-26T06:35:26.104879+0000","last_unstale":"2024-03-26T06:36:27.536655+0000","last_undegraded":"2024-03-26T06:36:27.536655+0000","last_fullsized":"2024-03-26T06:36:27.536655+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:44:57.444844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.533980+0000","last_change":"2024-03-26T06:35:27.133549+0000","last_active":"2024-03-26T06:36:27.533980+0000","last_peered":"2024-03-26T06:36:27.533980+0000","last_clean":"2024-03-26T06:36:27.533980+0000","last_became_active":"2024-03-26T06:35:27.132176+0000","last_became_peered":"2024-03-26T06:35:27.132176+0000","last_unstale":"2024-03-26T06:36:27.533980+0000","last_undegraded":"2024-03-26T06:36:27.533980+0000","last_fullsized":"2024-03-26T06:36:27.533980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:15:59.281268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536475+0000","last_change":"2024-03-26T06:35:26.101438+0000","last_active":"2024-03-26T06:36:27.536475+0000","last_peered":"2024-03-26T06:36:27.536475+0000","last_clean":"2024-03-26T06:36:27.536475+0000","last_became_active":"2024-03-26T06:35:26.101139+0000","last_became_peered":"2024-03-26T06:35:26.101139+0000","last_unstale":"2024-03-26T06:36:27.536475+0000","last_undegraded":"2024-03-26T06:36:27.536475+0000","last_fullsized":"2024-03-26T06:36:27.536475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:10:36.750046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535900+0000","last_change":"2024-03-26T06:35:27.127896+0000","last_active":"2024-03-26T06:36:27.535900+0000","last_peered":"2024-03-26T06:36:27.535900+0000","last_clean":"2024-03-26T06:36:27.535900+0000","last_became_active":"2024-03-26T06:35:27.127558+0000","last_became_peered":"2024-03-26T06:35:27.127558+0000","last_unstale":"2024-03-26T06:36:27.535900+0000","last_undegraded":"2024-03-26T06:36:27.535900+0000","last_fullsized":"2024-03-26T06:36:27.535900+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:04:58.033255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":99,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:43.281511+0000","last_change":"2024-03-26T06:35:26.100148+0000","last_active":"2024-03-26T06:36:43.281511+0000","last_peered":"2024-03-26T06:36:43.281511+0000","last_clean":"2024-03-26T06:36:43.281511+0000","last_became_active":"2024-03-26T06:35:26.099827+0000","last_became_peered":"2024-03-26T06:35:26.099827+0000","last_unstale":"2024-03-26T06:36:43.281511+0000","last_undegraded":"2024-03-26T06:36:43.281511+0000","last_fullsized":"2024-03-26T06:36:43.281511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:33:21.845866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":75,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536161+0000","last_change":"2024-03-26T06:35:27.125318+0000","last_active":"2024-03-26T06:36:27.536161+0000","last_peered":"2024-03-26T06:36:27.536161+0000","last_clean":"2024-03-26T06:36:27.536161+0000","last_became_active":"2024-03-26T06:35:27.125013+0000","last_became_peered":"2024-03-26T06:35:27.125013+0000","last_unstale":"2024-03-26T06:36:27.536161+0000","last_undegraded":"2024-03-26T06:36:27.536161+0000","last_fullsized":"2024-03-26T06:36:27.536161+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:50:25.123317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.532043+0000","last_change":"2024-03-26T06:35:26.097150+0000","last_active":"2024-03-26T06:36:27.532043+0000","last_peered":"2024-03-26T06:36:27.532043+0000","last_clean":"2024-03-26T06:36:27.532043+0000","last_became_active":"2024-03-26T06:35:26.096844+0000","last_became_peered":"2024-03-26T06:35:26.096844+0000","last_unstale":"2024-03-26T06:36:27.532043+0000","last_undegraded":"2024-03-26T06:36:27.532043+0000","last_fullsized":"2024-03-26T06:36:27.532043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:57:47.399493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535756+0000","last_change":"2024-03-26T06:35:27.132261+0000","last_active":"2024-03-26T06:36:27.535756+0000","last_peered":"2024-03-26T06:36:27.535756+0000","last_clean":"2024-03-26T06:36:27.535756+0000","last_became_active":"2024-03-26T06:35:27.131893+0000","last_became_peered":"2024-03-26T06:35:27.131893+0000","last_unstale":"2024-03-26T06:36:27.535756+0000","last_undegraded":"2024-03-26T06:36:27.535756+0000","last_fullsized":"2024-03-26T06:36:27.535756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:42.521515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562425+0000","last_change":"2024-03-26T06:35:26.092356+0000","last_active":"2024-03-26T06:36:27.562425+0000","last_peered":"2024-03-26T06:36:27.562425+0000","last_clean":"2024-03-26T06:36:27.562425+0000","last_became_active":"2024-03-26T06:35:26.091948+0000","last_became_peered":"2024-03-26T06:35:26.091948+0000","last_unstale":"2024-03-26T06:36:27.562425+0000","last_undegraded":"2024-03-26T06:36:27.562425+0000","last_fullsized":"2024-03-26T06:36:27.562425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:48:47.011884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540090+0000","last_change":"2024-03-26T06:35:27.129880+0000","last_active":"2024-03-26T06:36:27.540090+0000","last_peered":"2024-03-26T06:36:27.540090+0000","last_clean":"2024-03-26T06:36:27.540090+0000","last_became_active":"2024-03-26T06:35:27.129513+0000","last_became_peered":"2024-03-26T06:35:27.129513+0000","last_unstale":"2024-03-26T06:36:27.540090+0000","last_undegraded":"2024-03-26T06:36:27.540090+0000","last_fullsized":"2024-03-26T06:36:27.540090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:10:53.747657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535376+0000","last_change":"2024-03-26T06:35:26.095603+0000","last_active":"2024-03-26T06:36:27.535376+0000","last_peered":"2024-03-26T06:36:27.535376+0000","last_clean":"2024-03-26T06:36:27.535376+0000","last_became_active":"2024-03-26T06:35:26.095309+0000","last_became_peered":"2024-03-26T06:35:26.095309+0000","last_unstale":"2024-03-26T06:36:27.535376+0000","last_undegraded":"2024-03-26T06:36:27.535376+0000","last_fullsized":"2024-03-26T06:36:27.535376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:30:55.373598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.538485+0000","last_change":"2024-03-26T06:35:27.133271+0000","last_active":"2024-03-26T06:36:27.538485+0000","last_peered":"2024-03-26T06:36:27.538485+0000","last_clean":"2024-03-26T06:36:27.538485+0000","last_became_active":"2024-03-26T06:35:27.132553+0000","last_became_peered":"2024-03-26T06:35:27.132553+0000","last_unstale":"2024-03-26T06:36:27.538485+0000","last_undegraded":"2024-03-26T06:36:27.538485+0000","last_fullsized":"2024-03-26T06:36:27.538485+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:13:59.021131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:44.282278+0000","last_change":"2024-03-26T06:35:26.094323+0000","last_active":"2024-03-26T06:36:44.282278+0000","last_peered":"2024-03-26T06:36:44.282278+0000","last_clean":"2024-03-26T06:36:44.282278+0000","last_became_active":"2024-03-26T06:35:26.093814+0000","last_became_peered":"2024-03-26T06:35:26.093814+0000","last_unstale":"2024-03-26T06:36:44.282278+0000","last_undegraded":"2024-03-26T06:36:44.282278+0000","last_fullsized":"2024-03-26T06:36:44.282278+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:11.450427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534262+0000","last_change":"2024-03-26T06:35:27.127398+0000","last_active":"2024-03-26T06:36:27.534262+0000","last_peered":"2024-03-26T06:36:27.534262+0000","last_clean":"2024-03-26T06:36:27.534262+0000","last_became_active":"2024-03-26T06:35:27.127054+0000","last_became_peered":"2024-03-26T06:35:27.127054+0000","last_unstale":"2024-03-26T06:36:27.534262+0000","last_undegraded":"2024-03-26T06:36:27.534262+0000","last_fullsized":"2024-03-26T06:36:27.534262+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:51:04.150954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535969+0000","last_change":"2024-03-26T06:35:26.098531+0000","last_active":"2024-03-26T06:36:27.535969+0000","last_peered":"2024-03-26T06:36:27.535969+0000","last_clean":"2024-03-26T06:36:27.535969+0000","last_became_active":"2024-03-26T06:35:26.097942+0000","last_became_peered":"2024-03-26T06:35:26.097942+0000","last_unstale":"2024-03-26T06:36:27.535969+0000","last_undegraded":"2024-03-26T06:36:27.535969+0000","last_fullsized":"2024-03-26T06:36:27.535969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:44:59.884627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535814+0000","last_change":"2024-03-26T06:35:27.136145+0000","last_active":"2024-03-26T06:36:27.535814+0000","last_peered":"2024-03-26T06:36:27.535814+0000","last_clean":"2024-03-26T06:36:27.535814+0000","last_became_active":"2024-03-26T06:35:27.135560+0000","last_became_peered":"2024-03-26T06:35:27.135560+0000","last_unstale":"2024-03-26T06:36:27.535814+0000","last_undegraded":"2024-03-26T06:36:27.535814+0000","last_fullsized":"2024-03-26T06:36:27.535814+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:19:01.991175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536006+0000","last_change":"2024-03-26T06:35:26.095119+0000","last_active":"2024-03-26T06:36:27.536006+0000","last_peered":"2024-03-26T06:36:27.536006+0000","last_clean":"2024-03-26T06:36:27.536006+0000","last_became_active":"2024-03-26T06:35:26.094789+0000","last_became_peered":"2024-03-26T06:35:26.094789+0000","last_unstale":"2024-03-26T06:36:27.536006+0000","last_undegraded":"2024-03-26T06:36:27.536006+0000","last_fullsized":"2024-03-26T06:36:27.536006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:59:07.626207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534210+0000","last_change":"2024-03-26T06:35:27.133722+0000","last_active":"2024-03-26T06:36:27.534210+0000","last_peered":"2024-03-26T06:36:27.534210+0000","last_clean":"2024-03-26T06:36:27.534210+0000","last_became_active":"2024-03-26T06:35:27.132313+0000","last_became_peered":"2024-03-26T06:35:27.132313+0000","last_unstale":"2024-03-26T06:36:27.534210+0000","last_undegraded":"2024-03-26T06:36:27.534210+0000","last_fullsized":"2024-03-26T06:36:27.534210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:58:48.350708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'27","reported_seq":219,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:45.593235+0000","last_change":"2024-03-26T06:36:27.571382+0000","last_active":"2024-03-26T06:36:45.593235+0000","last_peered":"2024-03-26T06:36:45.593235+0000","last_clean":"2024-03-26T06:36:45.593235+0000","last_became_active":"2024-03-26T06:36:27.563425+0000","last_became_peered":"2024-03-26T06:36:27.563425+0000","last_unstale":"2024-03-26T06:36:45.593235+0000","last_undegraded":"2024-03-26T06:36:45.593235+0000","last_fullsized":"2024-03-26T06:36:45.593235+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:21:55.717347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59465,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":129,"num_read_kb":122,"num_write":28,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":59465,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536422+0000","last_change":"2024-03-26T06:35:27.127735+0000","last_active":"2024-03-26T06:36:27.536422+0000","last_peered":"2024-03-26T06:36:27.536422+0000","last_clean":"2024-03-26T06:36:27.536422+0000","last_became_active":"2024-03-26T06:35:27.127434+0000","last_became_peered":"2024-03-26T06:35:27.127434+0000","last_unstale":"2024-03-26T06:36:27.536422+0000","last_undegraded":"2024-03-26T06:36:27.536422+0000","last_fullsized":"2024-03-26T06:36:27.536422+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:30:09.745415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534120+0000","last_change":"2024-03-26T06:35:26.095943+0000","last_active":"2024-03-26T06:36:27.534120+0000","last_peered":"2024-03-26T06:36:27.534120+0000","last_clean":"2024-03-26T06:36:27.534120+0000","last_became_active":"2024-03-26T06:35:26.095577+0000","last_became_peered":"2024-03-26T06:35:26.095577+0000","last_unstale":"2024-03-26T06:36:27.534120+0000","last_undegraded":"2024-03-26T06:36:27.534120+0000","last_fullsized":"2024-03-26T06:36:27.534120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:42:34.459051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562209+0000","last_change":"2024-03-26T06:35:27.133199+0000","last_active":"2024-03-26T06:36:27.562209+0000","last_peered":"2024-03-26T06:36:27.562209+0000","last_clean":"2024-03-26T06:36:27.562209+0000","last_became_active":"2024-03-26T06:35:27.132187+0000","last_became_peered":"2024-03-26T06:35:27.132187+0000","last_unstale":"2024-03-26T06:36:27.562209+0000","last_undegraded":"2024-03-26T06:36:27.562209+0000","last_fullsized":"2024-03-26T06:36:27.562209+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:16:22.832804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535584+0000","last_change":"2024-03-26T06:35:26.100902+0000","last_active":"2024-03-26T06:36:27.535584+0000","last_peered":"2024-03-26T06:36:27.535584+0000","last_clean":"2024-03-26T06:36:27.535584+0000","last_became_active":"2024-03-26T06:35:26.100509+0000","last_became_peered":"2024-03-26T06:35:26.100509+0000","last_unstale":"2024-03-26T06:36:27.535584+0000","last_undegraded":"2024-03-26T06:36:27.535584+0000","last_fullsized":"2024-03-26T06:36:27.535584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:52:53.459610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535861+0000","last_change":"2024-03-26T06:35:27.129126+0000","last_active":"2024-03-26T06:36:27.535861+0000","last_peered":"2024-03-26T06:36:27.535861+0000","last_clean":"2024-03-26T06:36:27.535861+0000","last_became_active":"2024-03-26T06:35:27.128741+0000","last_became_peered":"2024-03-26T06:35:27.128741+0000","last_unstale":"2024-03-26T06:36:27.535861+0000","last_undegraded":"2024-03-26T06:36:27.535861+0000","last_fullsized":"2024-03-26T06:36:27.535861+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:09:15.613369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:40.279330+0000","last_change":"2024-03-26T06:35:26.096239+0000","last_active":"2024-03-26T06:36:40.279330+0000","last_peered":"2024-03-26T06:36:40.279330+0000","last_clean":"2024-03-26T06:36:40.279330+0000","last_became_active":"2024-03-26T06:35:26.095915+0000","last_became_peered":"2024-03-26T06:35:26.095915+0000","last_unstale":"2024-03-26T06:36:40.279330+0000","last_undegraded":"2024-03-26T06:36:40.279330+0000","last_fullsized":"2024-03-26T06:36:40.279330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:50:24.566322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535669+0000","last_change":"2024-03-26T06:35:27.125846+0000","last_active":"2024-03-26T06:36:27.535669+0000","last_peered":"2024-03-26T06:36:27.535669+0000","last_clean":"2024-03-26T06:36:27.535669+0000","last_became_active":"2024-03-26T06:35:27.125515+0000","last_became_peered":"2024-03-26T06:35:27.125515+0000","last_unstale":"2024-03-26T06:36:27.535669+0000","last_undegraded":"2024-03-26T06:36:27.535669+0000","last_fullsized":"2024-03-26T06:36:27.535669+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:02:17.358507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535873+0000","last_change":"2024-03-26T06:35:26.102857+0000","last_active":"2024-03-26T06:36:27.535873+0000","last_peered":"2024-03-26T06:36:27.535873+0000","last_clean":"2024-03-26T06:36:27.535873+0000","last_became_active":"2024-03-26T06:35:26.102552+0000","last_became_peered":"2024-03-26T06:35:26.102552+0000","last_unstale":"2024-03-26T06:36:27.535873+0000","last_undegraded":"2024-03-26T06:36:27.535873+0000","last_fullsized":"2024-03-26T06:36:27.535873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:31:30.833012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.537997+0000","last_change":"2024-03-26T06:36:27.537997+0000","last_active":"2024-03-26T06:36:27.537997+0000","last_peered":"2024-03-26T06:36:27.537997+0000","last_clean":"2024-03-26T06:36:27.537997+0000","last_became_active":"2024-03-26T06:36:27.537511+0000","last_became_peered":"2024-03-26T06:36:27.537511+0000","last_unstale":"2024-03-26T06:36:27.537997+0000","last_undegraded":"2024-03-26T06:36:27.537997+0000","last_fullsized":"2024-03-26T06:36:27.537997+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:45:11.644668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536538+0000","last_change":"2024-03-26T06:35:26.103876+0000","last_active":"2024-03-26T06:36:27.536538+0000","last_peered":"2024-03-26T06:36:27.536538+0000","last_clean":"2024-03-26T06:36:27.536538+0000","last_became_active":"2024-03-26T06:35:26.102981+0000","last_became_peered":"2024-03-26T06:35:26.102981+0000","last_unstale":"2024-03-26T06:36:27.536538+0000","last_undegraded":"2024-03-26T06:36:27.536538+0000","last_fullsized":"2024-03-26T06:36:27.536538+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:24:04.154212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531742+0000","last_change":"2024-03-26T06:35:27.127529+0000","last_active":"2024-03-26T06:36:27.531742+0000","last_peered":"2024-03-26T06:36:27.531742+0000","last_clean":"2024-03-26T06:36:27.531742+0000","last_became_active":"2024-03-26T06:35:27.127208+0000","last_became_peered":"2024-03-26T06:35:27.127208+0000","last_unstale":"2024-03-26T06:36:27.531742+0000","last_undegraded":"2024-03-26T06:36:27.531742+0000","last_fullsized":"2024-03-26T06:36:27.531742+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:29:35.313042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.538726+0000","last_change":"2024-03-26T06:35:26.099309+0000","last_active":"2024-03-26T06:36:27.538726+0000","last_peered":"2024-03-26T06:36:27.538726+0000","last_clean":"2024-03-26T06:36:27.538726+0000","last_became_active":"2024-03-26T06:35:26.098948+0000","last_became_peered":"2024-03-26T06:35:26.098948+0000","last_unstale":"2024-03-26T06:36:27.538726+0000","last_undegraded":"2024-03-26T06:36:27.538726+0000","last_fullsized":"2024-03-26T06:36:27.538726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:49:35.210018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.532317+0000","last_change":"2024-03-26T06:35:27.132441+0000","last_active":"2024-03-26T06:36:27.532317+0000","last_peered":"2024-03-26T06:36:27.532317+0000","last_clean":"2024-03-26T06:36:27.532317+0000","last_became_active":"2024-03-26T06:35:27.131846+0000","last_became_peered":"2024-03-26T06:35:27.131846+0000","last_unstale":"2024-03-26T06:36:27.532317+0000","last_undegraded":"2024-03-26T06:36:27.532317+0000","last_fullsized":"2024-03-26T06:36:27.532317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:14.779299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535824+0000","last_change":"2024-03-26T06:35:26.107214+0000","last_active":"2024-03-26T06:36:27.535824+0000","last_peered":"2024-03-26T06:36:27.535824+0000","last_clean":"2024-03-26T06:36:27.535824+0000","last_became_active":"2024-03-26T06:35:26.106935+0000","last_became_peered":"2024-03-26T06:35:26.106935+0000","last_unstale":"2024-03-26T06:36:27.535824+0000","last_undegraded":"2024-03-26T06:36:27.535824+0000","last_fullsized":"2024-03-26T06:36:27.535824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:49:33.101354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.532204+0000","last_change":"2024-03-26T06:35:27.131374+0000","last_active":"2024-03-26T06:36:27.532204+0000","last_peered":"2024-03-26T06:36:27.532204+0000","last_clean":"2024-03-26T06:36:27.532204+0000","last_became_active":"2024-03-26T06:35:27.131030+0000","last_became_peered":"2024-03-26T06:35:27.131030+0000","last_unstale":"2024-03-26T06:36:27.532204+0000","last_undegraded":"2024-03-26T06:36:27.532204+0000","last_fullsized":"2024-03-26T06:36:27.532204+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:33:40.809278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535514+0000","last_change":"2024-03-26T06:35:27.127240+0000","last_active":"2024-03-26T06:36:27.535514+0000","last_peered":"2024-03-26T06:36:27.535514+0000","last_clean":"2024-03-26T06:36:27.535514+0000","last_became_active":"2024-03-26T06:35:27.126923+0000","last_became_peered":"2024-03-26T06:35:27.126923+0000","last_unstale":"2024-03-26T06:36:27.535514+0000","last_undegraded":"2024-03-26T06:36:27.535514+0000","last_fullsized":"2024-03-26T06:36:27.535514+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:28:20.422536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":97,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:44.285390+0000","last_change":"2024-03-26T06:35:26.096718+0000","last_active":"2024-03-26T06:36:44.285390+0000","last_peered":"2024-03-26T06:36:44.285390+0000","last_clean":"2024-03-26T06:36:44.285390+0000","last_became_active":"2024-03-26T06:35:26.096407+0000","last_became_peered":"2024-03-26T06:35:26.096407+0000","last_unstale":"2024-03-26T06:36:44.285390+0000","last_undegraded":"2024-03-26T06:36:44.285390+0000","last_fullsized":"2024-03-26T06:36:44.285390+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:12:00.799567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":74,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540459+0000","last_change":"2024-03-26T06:35:27.138485+0000","last_active":"2024-03-26T06:36:27.540459+0000","last_peered":"2024-03-26T06:36:27.540459+0000","last_clean":"2024-03-26T06:36:27.540459+0000","last_became_active":"2024-03-26T06:35:27.138114+0000","last_became_peered":"2024-03-26T06:35:27.138114+0000","last_unstale":"2024-03-26T06:36:27.540459+0000","last_undegraded":"2024-03-26T06:36:27.540459+0000","last_fullsized":"2024-03-26T06:36:27.540459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:28:25.244683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562961+0000","last_change":"2024-03-26T06:35:26.096111+0000","last_active":"2024-03-26T06:36:27.562961+0000","last_peered":"2024-03-26T06:36:27.562961+0000","last_clean":"2024-03-26T06:36:27.562961+0000","last_became_active":"2024-03-26T06:35:26.095809+0000","last_became_peered":"2024-03-26T06:35:26.095809+0000","last_unstale":"2024-03-26T06:36:27.562961+0000","last_undegraded":"2024-03-26T06:36:27.562961+0000","last_fullsized":"2024-03-26T06:36:27.562961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:04:17.436961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"23'137","reported_seq":352,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535806+0000","last_change":"2024-03-26T06:34:13.886034+0000","last_active":"2024-03-26T06:36:27.535806+0000","last_peered":"2024-03-26T06:36:27.535806+0000","last_clean":"2024-03-26T06:36:27.535806+0000","last_became_active":"2024-03-26T06:34:13.885472+0000","last_became_peered":"2024-03-26T06:34:13.885472+0000","last_unstale":"2024-03-26T06:36:27.535806+0000","last_undegraded":"2024-03-26T06:36:27.535806+0000","last_fullsized":"2024-03-26T06:36:27.535806+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:30:52.854577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:30:52.854577+0000","last_clean_scrub_stamp":"2024-03-26T06:30:52.854577+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:43:35.315474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562322+0000","last_change":"2024-03-26T06:35:27.131778+0000","last_active":"2024-03-26T06:36:27.562322+0000","last_peered":"2024-03-26T06:36:27.562322+0000","last_clean":"2024-03-26T06:36:27.562322+0000","last_became_active":"2024-03-26T06:35:27.131433+0000","last_became_peered":"2024-03-26T06:35:27.131433+0000","last_unstale":"2024-03-26T06:36:27.562322+0000","last_undegraded":"2024-03-26T06:36:27.562322+0000","last_fullsized":"2024-03-26T06:36:27.562322+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:17:58.614371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535852+0000","last_change":"2024-03-26T06:35:26.106052+0000","last_active":"2024-03-26T06:36:27.535852+0000","last_peered":"2024-03-26T06:36:27.535852+0000","last_clean":"2024-03-26T06:36:27.535852+0000","last_became_active":"2024-03-26T06:35:26.105728+0000","last_became_peered":"2024-03-26T06:35:26.105728+0000","last_unstale":"2024-03-26T06:36:27.535852+0000","last_undegraded":"2024-03-26T06:36:27.535852+0000","last_fullsized":"2024-03-26T06:36:27.535852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:19:12.226026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536562+0000","last_change":"2024-03-26T06:35:26.099487+0000","last_active":"2024-03-26T06:36:27.536562+0000","last_peered":"2024-03-26T06:36:27.536562+0000","last_clean":"2024-03-26T06:36:27.536562+0000","last_became_active":"2024-03-26T06:35:26.099180+0000","last_became_peered":"2024-03-26T06:35:26.099180+0000","last_unstale":"2024-03-26T06:36:27.536562+0000","last_undegraded":"2024-03-26T06:36:27.536562+0000","last_fullsized":"2024-03-26T06:36:27.536562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:15:20.832451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562530+0000","last_change":"2024-03-26T06:35:27.125799+0000","last_active":"2024-03-26T06:36:27.562530+0000","last_peered":"2024-03-26T06:36:27.562530+0000","last_clean":"2024-03-26T06:36:27.562530+0000","last_became_active":"2024-03-26T06:35:27.125323+0000","last_became_peered":"2024-03-26T06:35:27.125323+0000","last_unstale":"2024-03-26T06:36:27.562530+0000","last_undegraded":"2024-03-26T06:36:27.562530+0000","last_fullsized":"2024-03-26T06:36:27.562530+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:07:31.162007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'8","reported_seq":39,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.549340+0000","last_change":"2024-03-26T06:36:27.549340+0000","last_active":"2024-03-26T06:36:27.549340+0000","last_peered":"2024-03-26T06:36:27.549340+0000","last_clean":"2024-03-26T06:36:27.549340+0000","last_became_active":"2024-03-26T06:36:27.541431+0000","last_became_peered":"2024-03-26T06:36:27.541431+0000","last_unstale":"2024-03-26T06:36:27.549340+0000","last_undegraded":"2024-03-26T06:36:27.549340+0000","last_fullsized":"2024-03-26T06:36:27.549340+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:53:43.126462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.541095+0000","last_change":"2024-03-26T06:35:27.128853+0000","last_active":"2024-03-26T06:36:27.541095+0000","last_peered":"2024-03-26T06:36:27.541095+0000","last_clean":"2024-03-26T06:36:27.541095+0000","last_became_active":"2024-03-26T06:35:27.128472+0000","last_became_peered":"2024-03-26T06:35:27.128472+0000","last_unstale":"2024-03-26T06:36:27.541095+0000","last_undegraded":"2024-03-26T06:36:27.541095+0000","last_fullsized":"2024-03-26T06:36:27.541095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:01:01.574623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562404+0000","last_change":"2024-03-26T06:35:26.100523+0000","last_active":"2024-03-26T06:36:27.562404+0000","last_peered":"2024-03-26T06:36:27.562404+0000","last_clean":"2024-03-26T06:36:27.562404+0000","last_became_active":"2024-03-26T06:35:26.099313+0000","last_became_peered":"2024-03-26T06:35:26.099313+0000","last_unstale":"2024-03-26T06:36:27.562404+0000","last_undegraded":"2024-03-26T06:36:27.562404+0000","last_fullsized":"2024-03-26T06:36:27.562404+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:43:44.182464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535841+0000","last_change":"2024-03-26T06:35:27.131438+0000","last_active":"2024-03-26T06:36:27.535841+0000","last_peered":"2024-03-26T06:36:27.535841+0000","last_clean":"2024-03-26T06:36:27.535841+0000","last_became_active":"2024-03-26T06:35:27.131112+0000","last_became_peered":"2024-03-26T06:35:27.131112+0000","last_unstale":"2024-03-26T06:36:27.535841+0000","last_undegraded":"2024-03-26T06:36:27.535841+0000","last_fullsized":"2024-03-26T06:36:27.535841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:05:07.621191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.533821+0000","last_change":"2024-03-26T06:35:26.105039+0000","last_active":"2024-03-26T06:36:27.533821+0000","last_peered":"2024-03-26T06:36:27.533821+0000","last_clean":"2024-03-26T06:36:27.533821+0000","last_became_active":"2024-03-26T06:35:26.104776+0000","last_became_peered":"2024-03-26T06:35:26.104776+0000","last_unstale":"2024-03-26T06:36:27.533821+0000","last_undegraded":"2024-03-26T06:36:27.533821+0000","last_fullsized":"2024-03-26T06:36:27.533821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T17:20:17.748144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536028+0000","last_change":"2024-03-26T06:35:27.126320+0000","last_active":"2024-03-26T06:36:27.536028+0000","last_peered":"2024-03-26T06:36:27.536028+0000","last_clean":"2024-03-26T06:36:27.536028+0000","last_became_active":"2024-03-26T06:35:27.125996+0000","last_became_peered":"2024-03-26T06:35:27.125996+0000","last_unstale":"2024-03-26T06:36:27.536028+0000","last_undegraded":"2024-03-26T06:36:27.536028+0000","last_fullsized":"2024-03-26T06:36:27.536028+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:08:21.386787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540194+0000","last_change":"2024-03-26T06:35:26.103303+0000","last_active":"2024-03-26T06:36:27.540194+0000","last_peered":"2024-03-26T06:36:27.540194+0000","last_clean":"2024-03-26T06:36:27.540194+0000","last_became_active":"2024-03-26T06:35:26.102311+0000","last_became_peered":"2024-03-26T06:35:26.102311+0000","last_unstale":"2024-03-26T06:36:27.540194+0000","last_undegraded":"2024-03-26T06:36:27.540194+0000","last_fullsized":"2024-03-26T06:36:27.540194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:44:13.845386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531613+0000","last_change":"2024-03-26T06:35:27.128845+0000","last_active":"2024-03-26T06:36:27.531613+0000","last_peered":"2024-03-26T06:36:27.531613+0000","last_clean":"2024-03-26T06:36:27.531613+0000","last_became_active":"2024-03-26T06:35:27.128532+0000","last_became_peered":"2024-03-26T06:35:27.128532+0000","last_unstale":"2024-03-26T06:36:27.531613+0000","last_undegraded":"2024-03-26T06:36:27.531613+0000","last_fullsized":"2024-03-26T06:36:27.531613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:35:08.087689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.533849+0000","last_change":"2024-03-26T06:35:26.107589+0000","last_active":"2024-03-26T06:36:27.533849+0000","last_peered":"2024-03-26T06:36:27.533849+0000","last_clean":"2024-03-26T06:36:27.533849+0000","last_became_active":"2024-03-26T06:35:26.107280+0000","last_became_peered":"2024-03-26T06:35:26.107280+0000","last_unstale":"2024-03-26T06:36:27.533849+0000","last_undegraded":"2024-03-26T06:36:27.533849+0000","last_fullsized":"2024-03-26T06:36:27.533849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:57:27.621484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540508+0000","last_change":"2024-03-26T06:35:27.131238+0000","last_active":"2024-03-26T06:36:27.540508+0000","last_peered":"2024-03-26T06:36:27.540508+0000","last_clean":"2024-03-26T06:36:27.540508+0000","last_became_active":"2024-03-26T06:35:27.130902+0000","last_became_peered":"2024-03-26T06:35:27.130902+0000","last_unstale":"2024-03-26T06:36:27.540508+0000","last_undegraded":"2024-03-26T06:36:27.540508+0000","last_fullsized":"2024-03-26T06:36:27.540508+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:02:47.180199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.567118+0000","last_change":"2024-03-26T06:36:27.567118+0000","last_active":"2024-03-26T06:36:27.567118+0000","last_peered":"2024-03-26T06:36:27.567118+0000","last_clean":"2024-03-26T06:36:27.567118+0000","last_became_active":"2024-03-26T06:36:27.563836+0000","last_became_peered":"2024-03-26T06:36:27.563836+0000","last_unstale":"2024-03-26T06:36:27.567118+0000","last_undegraded":"2024-03-26T06:36:27.567118+0000","last_fullsized":"2024-03-26T06:36:27.567118+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:21:00.417615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535830+0000","last_change":"2024-03-26T06:35:27.133273+0000","last_active":"2024-03-26T06:36:27.535830+0000","last_peered":"2024-03-26T06:36:27.535830+0000","last_clean":"2024-03-26T06:36:27.535830+0000","last_became_active":"2024-03-26T06:35:27.132955+0000","last_became_peered":"2024-03-26T06:35:27.132955+0000","last_unstale":"2024-03-26T06:36:27.535830+0000","last_undegraded":"2024-03-26T06:36:27.535830+0000","last_fullsized":"2024-03-26T06:36:27.535830+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:26:01.542473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534470+0000","last_change":"2024-03-26T06:35:26.103065+0000","last_active":"2024-03-26T06:36:27.534470+0000","last_peered":"2024-03-26T06:36:27.534470+0000","last_clean":"2024-03-26T06:36:27.534470+0000","last_became_active":"2024-03-26T06:35:26.102761+0000","last_became_peered":"2024-03-26T06:35:26.102761+0000","last_unstale":"2024-03-26T06:36:27.534470+0000","last_undegraded":"2024-03-26T06:36:27.534470+0000","last_fullsized":"2024-03-26T06:36:27.534470+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:09:27.650223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536071+0000","last_change":"2024-03-26T06:35:27.137820+0000","last_active":"2024-03-26T06:36:27.536071+0000","last_peered":"2024-03-26T06:36:27.536071+0000","last_clean":"2024-03-26T06:36:27.536071+0000","last_became_active":"2024-03-26T06:35:27.137375+0000","last_became_peered":"2024-03-26T06:35:27.137375+0000","last_unstale":"2024-03-26T06:36:27.536071+0000","last_undegraded":"2024-03-26T06:36:27.536071+0000","last_fullsized":"2024-03-26T06:36:27.536071+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:43:13.908662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535800+0000","last_change":"2024-03-26T06:35:26.094295+0000","last_active":"2024-03-26T06:36:27.535800+0000","last_peered":"2024-03-26T06:36:27.535800+0000","last_clean":"2024-03-26T06:36:27.535800+0000","last_became_active":"2024-03-26T06:35:26.093950+0000","last_became_peered":"2024-03-26T06:35:26.093950+0000","last_unstale":"2024-03-26T06:36:27.535800+0000","last_undegraded":"2024-03-26T06:36:27.535800+0000","last_fullsized":"2024-03-26T06:36:27.535800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:45:23.700237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535932+0000","last_change":"2024-03-26T06:35:27.131253+0000","last_active":"2024-03-26T06:36:27.535932+0000","last_peered":"2024-03-26T06:36:27.535932+0000","last_clean":"2024-03-26T06:36:27.535932+0000","last_became_active":"2024-03-26T06:35:27.130929+0000","last_became_peered":"2024-03-26T06:35:27.130929+0000","last_unstale":"2024-03-26T06:36:27.535932+0000","last_undegraded":"2024-03-26T06:36:27.535932+0000","last_fullsized":"2024-03-26T06:36:27.535932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:45:55.744139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562582+0000","last_change":"2024-03-26T06:35:26.104272+0000","last_active":"2024-03-26T06:36:27.562582+0000","last_peered":"2024-03-26T06:36:27.562582+0000","last_clean":"2024-03-26T06:36:27.562582+0000","last_became_active":"2024-03-26T06:35:26.103957+0000","last_became_peered":"2024-03-26T06:35:26.103957+0000","last_unstale":"2024-03-26T06:36:27.562582+0000","last_undegraded":"2024-03-26T06:36:27.562582+0000","last_fullsized":"2024-03-26T06:36:27.562582+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:25.074996+0000","last_clean_scrub_stamp":"2024-03-26T06:35:25.074996+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:09:29.037044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.536075+0000","last_change":"2024-03-26T06:35:27.128221+0000","last_active":"2024-03-26T06:36:27.536075+0000","last_peered":"2024-03-26T06:36:27.536075+0000","last_clean":"2024-03-26T06:36:27.536075+0000","last_became_active":"2024-03-26T06:35:27.126609+0000","last_became_peered":"2024-03-26T06:35:27.126609+0000","last_unstale":"2024-03-26T06:36:27.536075+0000","last_undegraded":"2024-03-26T06:36:27.536075+0000","last_fullsized":"2024-03-26T06:36:27.536075+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:59:56.314853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.539868+0000","last_change":"2024-03-26T06:36:27.539868+0000","last_active":"2024-03-26T06:36:27.539868+0000","last_peered":"2024-03-26T06:36:27.539868+0000","last_clean":"2024-03-26T06:36:27.539868+0000","last_became_active":"2024-03-26T06:36:27.539468+0000","last_became_peered":"2024-03-26T06:36:27.539468+0000","last_unstale":"2024-03-26T06:36:27.539868+0000","last_undegraded":"2024-03-26T06:36:27.539868+0000","last_fullsized":"2024-03-26T06:36:27.539868+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:56:33.174588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534265+0000","last_change":"2024-03-26T06:35:27.127975+0000","last_active":"2024-03-26T06:36:27.534265+0000","last_peered":"2024-03-26T06:36:27.534265+0000","last_clean":"2024-03-26T06:36:27.534265+0000","last_became_active":"2024-03-26T06:35:27.127635+0000","last_became_peered":"2024-03-26T06:35:27.127635+0000","last_unstale":"2024-03-26T06:36:27.534265+0000","last_undegraded":"2024-03-26T06:36:27.534265+0000","last_fullsized":"2024-03-26T06:36:27.534265+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:48.587117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.533820+0000","last_change":"2024-03-26T06:35:27.133538+0000","last_active":"2024-03-26T06:36:27.533820+0000","last_peered":"2024-03-26T06:36:27.533820+0000","last_clean":"2024-03-26T06:36:27.533820+0000","last_became_active":"2024-03-26T06:35:27.133104+0000","last_became_peered":"2024-03-26T06:35:27.133104+0000","last_unstale":"2024-03-26T06:36:27.533820+0000","last_undegraded":"2024-03-26T06:36:27.533820+0000","last_fullsized":"2024-03-26T06:36:27.533820+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:20:10.417567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535554+0000","last_change":"2024-03-26T06:35:27.129503+0000","last_active":"2024-03-26T06:36:27.535554+0000","last_peered":"2024-03-26T06:36:27.535554+0000","last_clean":"2024-03-26T06:36:27.535554+0000","last_became_active":"2024-03-26T06:35:27.128930+0000","last_became_peered":"2024-03-26T06:35:27.128930+0000","last_unstale":"2024-03-26T06:36:27.535554+0000","last_undegraded":"2024-03-26T06:36:27.535554+0000","last_fullsized":"2024-03-26T06:36:27.535554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:23:37.476593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562432+0000","last_change":"2024-03-26T06:35:27.137385+0000","last_active":"2024-03-26T06:36:27.562432+0000","last_peered":"2024-03-26T06:36:27.562432+0000","last_clean":"2024-03-26T06:36:27.562432+0000","last_became_active":"2024-03-26T06:35:27.136969+0000","last_became_peered":"2024-03-26T06:35:27.136969+0000","last_unstale":"2024-03-26T06:36:27.562432+0000","last_undegraded":"2024-03-26T06:36:27.562432+0000","last_fullsized":"2024-03-26T06:36:27.562432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:40:16.722381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531790+0000","last_change":"2024-03-26T06:35:27.125287+0000","last_active":"2024-03-26T06:36:27.531790+0000","last_peered":"2024-03-26T06:36:27.531790+0000","last_clean":"2024-03-26T06:36:27.531790+0000","last_became_active":"2024-03-26T06:35:27.124274+0000","last_became_peered":"2024-03-26T06:35:27.124274+0000","last_unstale":"2024-03-26T06:36:27.531790+0000","last_undegraded":"2024-03-26T06:36:27.531790+0000","last_fullsized":"2024-03-26T06:36:27.531790+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:33:43.461949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534081+0000","last_change":"2024-03-26T06:35:27.124805+0000","last_active":"2024-03-26T06:36:27.534081+0000","last_peered":"2024-03-26T06:36:27.534081+0000","last_clean":"2024-03-26T06:36:27.534081+0000","last_became_active":"2024-03-26T06:35:27.123947+0000","last_became_peered":"2024-03-26T06:35:27.123947+0000","last_unstale":"2024-03-26T06:36:27.534081+0000","last_undegraded":"2024-03-26T06:36:27.534081+0000","last_fullsized":"2024-03-26T06:36:27.534081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:43:47.273083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540556+0000","last_change":"2024-03-26T06:35:27.141167+0000","last_active":"2024-03-26T06:36:27.540556+0000","last_peered":"2024-03-26T06:36:27.540556+0000","last_clean":"2024-03-26T06:36:27.540556+0000","last_became_active":"2024-03-26T06:35:27.140781+0000","last_became_peered":"2024-03-26T06:35:27.140781+0000","last_unstale":"2024-03-26T06:36:27.540556+0000","last_undegraded":"2024-03-26T06:36:27.540556+0000","last_fullsized":"2024-03-26T06:36:27.540556+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:42:36.705671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.535401+0000","last_change":"2024-03-26T06:35:27.132336+0000","last_active":"2024-03-26T06:36:27.535401+0000","last_peered":"2024-03-26T06:36:27.535401+0000","last_clean":"2024-03-26T06:36:27.535401+0000","last_became_active":"2024-03-26T06:35:27.131944+0000","last_became_peered":"2024-03-26T06:35:27.131944+0000","last_unstale":"2024-03-26T06:36:27.535401+0000","last_undegraded":"2024-03-26T06:36:27.535401+0000","last_fullsized":"2024-03-26T06:36:27.535401+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:51:57.916873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.540170+0000","last_change":"2024-03-26T06:35:27.141297+0000","last_active":"2024-03-26T06:36:27.540170+0000","last_peered":"2024-03-26T06:36:27.540170+0000","last_clean":"2024-03-26T06:36:27.540170+0000","last_became_active":"2024-03-26T06:35:27.140932+0000","last_became_peered":"2024-03-26T06:35:27.140932+0000","last_unstale":"2024-03-26T06:36:27.540170+0000","last_undegraded":"2024-03-26T06:36:27.540170+0000","last_fullsized":"2024-03-26T06:36:27.540170+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:02:17.942885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.534003+0000","last_change":"2024-03-26T06:35:27.135381+0000","last_active":"2024-03-26T06:36:27.534003+0000","last_peered":"2024-03-26T06:36:27.534003+0000","last_clean":"2024-03-26T06:36:27.534003+0000","last_became_active":"2024-03-26T06:35:27.135053+0000","last_became_peered":"2024-03-26T06:35:27.135053+0000","last_unstale":"2024-03-26T06:36:27.534003+0000","last_undegraded":"2024-03-26T06:36:27.534003+0000","last_fullsized":"2024-03-26T06:36:27.534003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:40:21.960182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.562542+0000","last_change":"2024-03-26T06:35:27.130780+0000","last_active":"2024-03-26T06:36:27.562542+0000","last_peered":"2024-03-26T06:36:27.562542+0000","last_clean":"2024-03-26T06:36:27.562542+0000","last_became_active":"2024-03-26T06:35:27.130171+0000","last_became_peered":"2024-03-26T06:35:27.130171+0000","last_unstale":"2024-03-26T06:36:27.562542+0000","last_undegraded":"2024-03-26T06:36:27.562542+0000","last_fullsized":"2024-03-26T06:36:27.562542+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:53:18.053617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T06:36:27.531641+0000","last_change":"2024-03-26T06:35:27.129677+0000","last_active":"2024-03-26T06:36:27.531641+0000","last_peered":"2024-03-26T06:36:27.531641+0000","last_clean":"2024-03-26T06:36:27.531641+0000","last_became_active":"2024-03-26T06:35:27.129275+0000","last_became_peered":"2024-03-26T06:35:27.129275+0000","last_unstale":"2024-03-26T06:36:27.531641+0000","last_undegraded":"2024-03-26T06:36:27.531641+0000","last_fullsized":"2024-03-26T06:36:27.531641+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T06:35:26.081948+0000","last_clean_scrub_stamp":"2024-03-26T06:35:26.081948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:25:18.851198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63466,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":439,"num_read_kb":289,"num_write":155,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":60047,"num_keys_recovered":7,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":190398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351005,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":962560,"data_stored":459933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612642,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28636,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716516,"statfs":{"total":95995035648,"available":95965712384,"internally_reserved":0,"allocated":1654784,"data_stored":1110688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":459587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037804,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36296,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708856,"statfs":{"total":95995035648,"available":95957868544,"internally_reserved":0,"allocated":974848,"data_stored":459507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299441,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32712,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712440,"statfs":{"total":95995035648,"available":95961538560,"internally_reserved":0,"allocated":1564672,"data_stored":1050855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462967,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":458976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574927,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1024000,"data_stored":518234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738452,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":460247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411403,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":40460,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93704692,"statfs":{"total":95995035648,"available":95953604608,"internally_reserved":0,"allocated":978944,"data_stored":460119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149765,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":978944,"data_stored":459287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888128,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1036288,"data_stored":519269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724603,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1581056,"data_stored":1050730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T06:36:47.514 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T06:36:47.514 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T06:36:47.514 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T06:36:47.514 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T06:36:47.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:47 smithi121 ceph-mon[36349]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:47.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:47 smithi121 ceph-mon[36349]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:36:47.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:47 smithi170 ceph-mon[36912]: pgmap v328: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:47.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:47 smithi170 ceph-mon[36912]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T06:36:47.760 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:49.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.459+0000 7f6f77fcf700 1 -- 172.21.15.38:0/2448415119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f70107170 msgr2=0x7f6f7010ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:49.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.459+0000 7f6f77fcf700 1 --2- 172.21.15.38:0/2448415119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f70107170 0x7f6f7010ecf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6f6800b0d0 tx=0x7f6f6800b3e0 comp rx=0 tx=0).stop 2024-03-26T06:36:49.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 -- 172.21.15.38:0/2448415119 shutdown_connections 2024-03-26T06:36:49.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 --2- 172.21.15.38:0/2448415119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f70107170 0x7f6f7010ecf0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:49.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 --2- 172.21.15.38:0/2448415119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f701020c0 0x7f6f701024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:49.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 --2- 172.21.15.38:0/2448415119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f70102a70 0x7f6f70106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:49.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 -- 172.21.15.38:0/2448415119 >> 172.21.15.38:0/2448415119 conn(0x7f6f70078f00 msgr2=0x7f6f70079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:49.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 -- 172.21.15.38:0/2448415119 shutdown_connections 2024-03-26T06:36:49.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 -- 172.21.15.38:0/2448415119 wait complete. 2024-03-26T06:36:49.467 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:49 smithi038 ceph-mon[32265]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:49.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 Processor -- start 2024-03-26T06:36:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 -- start start 2024-03-26T06:36:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f701020c0 0x7f6f700740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f70102a70 0x7f6f70074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.460+0000 7f6f77fcf700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f70107170 0x7f6f70074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f77fcf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70112110 con 0x7f6f70102a70 2024-03-26T06:36:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f77fcf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70111f90 con 0x7f6f701020c0 2024-03-26T06:36:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f77fcf700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70112290 con 0x7f6f70107170 2024-03-26T06:36:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f7556a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f70102a70 0x7f6f70074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f7556a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f70102a70 0x7f6f70074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43124/0 (socket says 172.21.15.38:43124) 2024-03-26T06:36:49.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f7556a700 1 -- 172.21.15.38:0/3275139748 learned_addr learned my addr 172.21.15.38:0/3275139748 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f75d6b700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f701020c0 0x7f6f700740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f7656c700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f70107170 0x7f6f70074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f7556a700 1 -- 172.21.15.38:0/3275139748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f70107170 msgr2=0x7f6f70074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f7556a700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f70107170 0x7f6f70074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f7556a700 1 -- 172.21.15.38:0/3275139748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f701020c0 msgr2=0x7f6f700740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f7556a700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f701020c0 0x7f6f700740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f7556a700 1 -- 172.21.15.38:0/3275139748 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f701a69f0 con 0x7f6f70102a70 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f7556a700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f70102a70 0x7f6f70074620 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000c8c0 tx=0x7f6f6000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f66ffd700 1 -- 172.21.15.38:0/3275139748 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60004220 con 0x7f6f70102a70 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f66ffd700 1 -- 172.21.15.38:0/3275139748 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f60007de0 con 0x7f6f70102a70 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f66ffd700 1 -- 172.21.15.38:0/3275139748 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f60005020 con 0x7f6f70102a70 2024-03-26T06:36:49.471 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.461+0000 7f6f77fcf700 1 -- 172.21.15.38:0/3275139748 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f701a6ce0 con 0x7f6f70102a70 2024-03-26T06:36:49.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.462+0000 7f6f77fcf700 1 -- 172.21.15.38:0/3275139748 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f701a7210 con 0x7f6f70102a70 2024-03-26T06:36:49.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.463+0000 7f6f77fcf700 1 -- 172.21.15.38:0/3275139748 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f70068aa0 con 0x7f6f70102a70 2024-03-26T06:36:49.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.464+0000 7f6f66ffd700 1 -- 172.21.15.38:0/3275139748 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f6f60007900 con 0x7f6f70102a70 2024-03-26T06:36:49.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.464+0000 7f6f66ffd700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6f5c0a29a0 0x7f6f5c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:49.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.464+0000 7f6f75d6b700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6f5c0a29a0 0x7f6f5c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:49.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.464+0000 7f6f66ffd700 1 -- 172.21.15.38:0/3275139748 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f6f600ce6d0 con 0x7f6f70102a70 2024-03-26T06:36:49.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.465+0000 7f6f75d6b700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6f5c0a29a0 0x7f6f5c0a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c005ea0 tx=0x7f6f6c005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:49.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.469+0000 7f6f66ffd700 1 -- 172.21.15.38:0/3275139748 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6f60014030 con 0x7f6f70102a70 2024-03-26T06:36:49.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:49 smithi121 ceph-mon[36349]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:49.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:49 smithi170 ceph-mon[36912]: pgmap v329: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:49.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.715+0000 7f6f77fcf700 1 -- 172.21.15.38:0/3275139748 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6f7002d830 con 0x7f6f70102a70 2024-03-26T06:36:49.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.716+0000 7f6f66ffd700 1 -- 172.21.15.38:0/3275139748 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f6f6009b610 con 0x7f6f70102a70 2024-03-26T06:36:49.718 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:36:49.718 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T06:36:49.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 -- 172.21.15.38:0/3275139748 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6f5c0a29a0 msgr2=0x7f6f5c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:49.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6f5c0a29a0 0x7f6f5c0a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c005ea0 tx=0x7f6f6c005e30 comp rx=0 tx=0).stop 2024-03-26T06:36:49.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 -- 172.21.15.38:0/3275139748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f70102a70 msgr2=0x7f6f70074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:49.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f70102a70 0x7f6f70074620 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6f6000c8c0 tx=0x7f6f6000cc80 comp rx=0 tx=0).stop 2024-03-26T06:36:49.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 -- 172.21.15.38:0/3275139748 shutdown_connections 2024-03-26T06:36:49.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f70107170 0x7f6f70074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:49.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6f5c0a29a0 0x7f6f5c0a4e60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:49.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f701020c0 0x7f6f700740e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:49.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 --2- 172.21.15.38:0/3275139748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f70102a70 0x7f6f70074620 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:49.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 -- 172.21.15.38:0/3275139748 >> 172.21.15.38:0/3275139748 conn(0x7f6f70078f00 msgr2=0x7f6f700ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:49.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 -- 172.21.15.38:0/3275139748 shutdown_connections 2024-03-26T06:36:49.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:49.718+0000 7f6f77fcf700 1 -- 172.21.15.38:0/3275139748 wait complete. 2024-03-26T06:36:50.373 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T06:36:50.373 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:36:50.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.477+0000 7f7de20fc640 1 -- 172.21.15.38:0/908602746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ddc101400 msgr2=0x7f7ddc101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:50.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.477+0000 7f7de20fc640 1 --2- 172.21.15.38:0/908602746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ddc101400 0x7f7ddc101880 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0009900 tx=0x7f7dd002bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:50.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.477+0000 7f7de20fc640 1 -- 172.21.15.38:0/908602746 shutdown_connections 2024-03-26T06:36:50.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.477+0000 7f7de20fc640 1 --2- 172.21.15.38:0/908602746 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ddc101dc0 0x7f7ddc108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.477+0000 7f7de20fc640 1 --2- 172.21.15.38:0/908602746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ddc101400 0x7f7ddc101880 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.477+0000 7f7de20fc640 1 --2- 172.21.15.38:0/908602746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ddc100210 0x7f7ddc100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.477+0000 7f7de20fc640 1 -- 172.21.15.38:0/908602746 >> 172.21.15.38:0/908602746 conn(0x7f7ddc0fb780 msgr2=0x7f7ddc0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.477+0000 7f7de20fc640 1 -- 172.21.15.38:0/908602746 shutdown_connections 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.477+0000 7f7de20fc640 1 -- 172.21.15.38:0/908602746 wait complete. 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.477+0000 7f7de20fc640 1 Processor -- start 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7de20fc640 1 -- start start 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7de20fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ddc100210 0x7f7ddc068fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7de20fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ddc101400 0x7f7ddc19c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7de20fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ddc101dc0 0x7f7ddc19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7de20fc640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc10bf50 con 0x7f7ddc100210 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7de20fc640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc10bdd0 con 0x7f7ddc101400 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7de20fc640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc10c0d0 con 0x7f7ddc101dc0 2024-03-26T06:36:50.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7ddb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ddc100210 0x7f7ddc068fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7ddb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ddc100210 0x7f7ddc068fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43134/0 (socket says 172.21.15.38:43134) 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7ddb7fe640 1 -- 172.21.15.38:0/2897781542 learned_addr learned my addr 172.21.15.38:0/2897781542 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7ddb7fe640 1 -- 172.21.15.38:0/2897781542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ddc101dc0 msgr2=0x7f7ddc19e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7ddb7fe640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ddc101dc0 0x7f7ddc19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7ddb7fe640 1 -- 172.21.15.38:0/2897781542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ddc101400 msgr2=0x7f7ddc19c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7ddaffd640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ddc101400 0x7f7ddc19c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7ddb7fe640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ddc101400 0x7f7ddc19c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7ddb7fe640 1 -- 172.21.15.38:0/2897781542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ddc19f2e0 con 0x7f7ddc100210 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7ddb7fe640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ddc100210 0x7f7ddc068fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00b450 tx=0x7f7dcc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7dd8ff9640 1 -- 172.21.15.38:0/2897781542 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc015020 con 0x7f7ddc100210 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7de20fc640 1 -- 172.21.15.38:0/2897781542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ddc19f570 con 0x7f7ddc100210 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7dd8ff9640 1 -- 172.21.15.38:0/2897781542 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7dcc011d70 con 0x7f7ddc100210 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7dd8ff9640 1 -- 172.21.15.38:0/2897781542 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc00f340 con 0x7f7ddc100210 2024-03-26T06:36:50.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.478+0000 7f7de20fc640 1 -- 172.21.15.38:0/2897781542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ddc19fac0 con 0x7f7ddc100210 2024-03-26T06:36:50.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.479+0000 7f7de20fc640 1 -- 172.21.15.38:0/2897781542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ddc1074d0 con 0x7f7ddc100210 2024-03-26T06:36:50.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.481+0000 7f7dd8ff9640 1 -- 172.21.15.38:0/2897781542 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7dcc018020 con 0x7f7ddc100210 2024-03-26T06:36:50.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.482+0000 7f7dd8ff9640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7dc40a2890 0x7f7dc40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:50.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.482+0000 7f7ddaffd640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7dc40a2890 0x7f7dc40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:50.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.482+0000 7f7dd8ff9640 1 -- 172.21.15.38:0/2897781542 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f7dcc0ceb80 con 0x7f7ddc100210 2024-03-26T06:36:50.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.482+0000 7f7ddaffd640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7dc40a2890 0x7f7dc40a4d50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00096c0 tx=0x7f7dd0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:50.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.485+0000 7f7dd8ff9640 1 -- 172.21.15.38:0/2897781542 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7dcc01f020 con 0x7f7ddc100210 2024-03-26T06:36:50.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:50 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3275139748' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T06:36:50.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:50 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3275139748' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T06:36:50.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.668+0000 7f7de20fc640 1 -- 172.21.15.38:0/2897781542 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ddc02d0c0 con 0x7f7ddc100210 2024-03-26T06:36:50.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.670+0000 7f7dd8ff9640 1 -- 172.21.15.38:0/2897781542 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22634 (secure 0 0 0) 0x7f7dcc00d040 con 0x7f7ddc100210 2024-03-26T06:36:50.672 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:36:50.673 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":88,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:36:27.512655+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:36:50.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 -- 172.21.15.38:0/2897781542 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7dc40a2890 msgr2=0x7f7dc40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:50.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7dc40a2890 0x7f7dc40a4d50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00096c0 tx=0x7f7dd0005fd0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 -- 172.21.15.38:0/2897781542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ddc100210 msgr2=0x7f7ddc068fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:50.674 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ddc100210 0x7f7ddc068fc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc00b450 tx=0x7f7dcc00b910 comp rx=0 tx=0).stop 2024-03-26T06:36:50.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 -- 172.21.15.38:0/2897781542 shutdown_connections 2024-03-26T06:36:50.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f7dc40a2890 0x7f7dc40a4d50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7ddc101dc0 0x7f7ddc19e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7ddc101400 0x7f7ddc19c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 --2- 172.21.15.38:0/2897781542 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7ddc100210 0x7f7ddc068fc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 -- 172.21.15.38:0/2897781542 >> 172.21.15.38:0/2897781542 conn(0x7f7ddc0fb780 msgr2=0x7f7ddc1046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:50.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 -- 172.21.15.38:0/2897781542 shutdown_connections 2024-03-26T06:36:50.675 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.672+0000 7f7de20fc640 1 -- 172.21.15.38:0/2897781542 wait complete. 2024-03-26T06:36:50.684 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:50 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3275139748' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T06:36:50.686 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T06:30:52.485897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-26T06:35:24.501259+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-26T06:35:25.675299+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T06:36:50.686 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T06:36:50.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.789+0000 7f17f7ad1640 1 -- 172.21.15.38:0/4254819811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f0101d70 msgr2=0x7f17f0104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:50.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.789+0000 7f17f7ad1640 1 --2- 172.21.15.38:0/4254819811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f0101d70 0x7f17f0104160 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009900 tx=0x7f17ec02bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:50.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.789+0000 7f17f7ad1640 1 -- 172.21.15.38:0/4254819811 shutdown_connections 2024-03-26T06:36:50.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.789+0000 7f17f7ad1640 1 --2- 172.21.15.38:0/4254819811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17f0106fd0 0x7f17f0109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.789+0000 7f17f7ad1640 1 --2- 172.21.15.38:0/4254819811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f01046a0 0x7f17f0106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.789+0000 7f17f7ad1640 1 --2- 172.21.15.38:0/4254819811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f0101d70 0x7f17f0104160 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.789+0000 7f17f7ad1640 1 -- 172.21.15.38:0/4254819811 >> 172.21.15.38:0/4254819811 conn(0x7f17f00fb780 msgr2=0x7f17f00fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.789+0000 7f17f7ad1640 1 -- 172.21.15.38:0/4254819811 shutdown_connections 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.789+0000 7f17f7ad1640 1 -- 172.21.15.38:0/4254819811 wait complete. 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f7ad1640 1 Processor -- start 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f7ad1640 1 -- start start 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f7ad1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f0101d70 0x7f17f0197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f7ad1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f01046a0 0x7f17f0197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f7ad1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17f0106fd0 0x7f17f019ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f7ad1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f010cf80 con 0x7f17f01046a0 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f7ad1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f010ce00 con 0x7f17f0101d70 2024-03-26T06:36:50.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f7ad1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17f010d100 con 0x7f17f0106fd0 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f506c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f01046a0 0x7f17f0197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f506c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f01046a0 0x7f17f0197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43156/0 (socket says 172.21.15.38:43156) 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f506c640 1 -- 172.21.15.38:0/3996052268 learned_addr learned my addr 172.21.15.38:0/3996052268 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f586d640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f0101d70 0x7f17f0197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f606e640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17f0106fd0 0x7f17f019ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f506c640 1 -- 172.21.15.38:0/3996052268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17f0106fd0 msgr2=0x7f17f019ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f506c640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17f0106fd0 0x7f17f019ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f506c640 1 -- 172.21.15.38:0/3996052268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f0101d70 msgr2=0x7f17f0197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f506c640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f0101d70 0x7f17f0197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f506c640 1 -- 172.21.15.38:0/3996052268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17f019f270 con 0x7f17f01046a0 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f586d640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f0101d70 0x7f17f0197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f506c640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f01046a0 0x7f17f0197b90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f17e0005fd0 tx=0x7f17e000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:50.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17e6ffd640 1 -- 172.21.15.38:0/3996052268 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0017070 con 0x7f17f01046a0 2024-03-26T06:36:50.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.790+0000 7f17f7ad1640 1 -- 172.21.15.38:0/3996052268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17f019f560 con 0x7f17f01046a0 2024-03-26T06:36:50.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.791+0000 7f17e6ffd640 1 -- 172.21.15.38:0/3996052268 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17e0009c30 con 0x7f17f01046a0 2024-03-26T06:36:50.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.791+0000 7f17e6ffd640 1 -- 172.21.15.38:0/3996052268 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0010460 con 0x7f17f01046a0 2024-03-26T06:36:50.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.791+0000 7f17f7ad1640 1 -- 172.21.15.38:0/3996052268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f019fb00 con 0x7f17f01046a0 2024-03-26T06:36:50.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.791+0000 7f17f7ad1640 1 -- 172.21.15.38:0/3996052268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17f00fd270 con 0x7f17f01046a0 2024-03-26T06:36:50.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.792+0000 7f17e6ffd640 1 -- 172.21.15.38:0/3996052268 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f17e0003e70 con 0x7f17f01046a0 2024-03-26T06:36:50.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.793+0000 7f17e6ffd640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f17dc0a2890 0x7f17dc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:50.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.793+0000 7f17e6ffd640 1 -- 172.21.15.38:0/3996052268 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f17e00ce9d0 con 0x7f17f01046a0 2024-03-26T06:36:50.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.793+0000 7f17f586d640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f17dc0a2890 0x7f17dc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:50.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.793+0000 7f17f586d640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f17dc0a2890 0x7f17dc0a4d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f17ec006010 tx=0x7f17ec02c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:50.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.798+0000 7f17e6ffd640 1 -- 172.21.15.38:0/3996052268 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f17e0020030 con 0x7f17f01046a0 2024-03-26T06:36:50.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.981+0000 7f17f7ad1640 1 -- 172.21.15.38:0/3996052268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17f002d070 con 0x7f17f01046a0 2024-03-26T06:36:50.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.981+0000 7f17e6ffd640 1 -- 172.21.15.38:0/3996052268 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17e009b910 con 0x7f17f01046a0 2024-03-26T06:36:50.984 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-03-26T06:36:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.983+0000 7f17f7ad1640 1 -- 172.21.15.38:0/3996052268 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f17dc0a2890 msgr2=0x7f17dc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.983+0000 7f17f7ad1640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f17dc0a2890 0x7f17dc0a4d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f17ec006010 tx=0x7f17ec02c8a0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.983+0000 7f17f7ad1640 1 -- 172.21.15.38:0/3996052268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f01046a0 msgr2=0x7f17f0197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.983+0000 7f17f7ad1640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f01046a0 0x7f17f0197b90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f17e0005fd0 tx=0x7f17e000dde0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.983+0000 7f17f7ad1640 1 -- 172.21.15.38:0/3996052268 shutdown_connections 2024-03-26T06:36:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.983+0000 7f17f7ad1640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f17dc0a2890 0x7f17dc0a4d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.983+0000 7f17f7ad1640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f17f0106fd0 0x7f17f019ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.983+0000 7f17f7ad1640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f17f01046a0 0x7f17f0197b90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.983+0000 7f17f7ad1640 1 --2- 172.21.15.38:0/3996052268 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17f0101d70 0x7f17f0197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.983+0000 7f17f7ad1640 1 -- 172.21.15.38:0/3996052268 >> 172.21.15.38:0/3996052268 conn(0x7f17f00fb780 msgr2=0x7f17f0102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.984+0000 7f17f7ad1640 1 -- 172.21.15.38:0/3996052268 shutdown_connections 2024-03-26T06:36:50.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:50.984+0000 7f17f7ad1640 1 -- 172.21.15.38:0/3996052268 wait complete. 2024-03-26T06:36:50.997 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-26T06:36:51.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.100+0000 7f693cf09640 1 -- 172.21.15.38:0/190679585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6938101550 msgr2=0x7f6938107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.100+0000 7f693cf09640 1 --2- 172.21.15.38:0/190679585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6938101550 0x7f6938107de0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f693000aa10 tx=0x7f693002cf70 comp rx=0 tx=0).stop 2024-03-26T06:36:51.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.100+0000 7f693cf09640 1 -- 172.21.15.38:0/190679585 shutdown_connections 2024-03-26T06:36:51.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.100+0000 7f693cf09640 1 --2- 172.21.15.38:0/190679585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6938101550 0x7f6938107de0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.100+0000 7f693cf09640 1 --2- 172.21.15.38:0/190679585 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6938100b90 0x7f6938101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.100+0000 7f693cf09640 1 --2- 172.21.15.38:0/190679585 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69380ff9a0 0x7f69380ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.100+0000 7f693cf09640 1 -- 172.21.15.38:0/190679585 >> 172.21.15.38:0/190679585 conn(0x7f69380faef0 msgr2=0x7f69380fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:51.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.100+0000 7f693cf09640 1 -- 172.21.15.38:0/190679585 shutdown_connections 2024-03-26T06:36:51.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.100+0000 7f693cf09640 1 -- 172.21.15.38:0/190679585 wait complete. 2024-03-26T06:36:51.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.100+0000 7f693cf09640 1 Processor -- start 2024-03-26T06:36:51.105 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f693cf09640 1 -- start start 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f693cf09640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69380ff9a0 0x7f6938197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f693cf09640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6938100b90 0x7f6938197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f693cf09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6938101550 0x7f693819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f693cf09640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693810adb0 con 0x7f6938101550 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f693cf09640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693810ac30 con 0x7f69380ff9a0 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f693cf09640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693810af30 con 0x7f6938100b90 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f6936d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6938101550 0x7f693819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f6936d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6938101550 0x7f693819ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43176/0 (socket says 172.21.15.38:43176) 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f6936d9d640 1 -- 172.21.15.38:0/1511585830 learned_addr learned my addr 172.21.15.38:0/1511585830 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f6935d9b640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6938100b90 0x7f6938197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f6936d9d640 1 -- 172.21.15.38:0/1511585830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6938100b90 msgr2=0x7f6938197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f6936d9d640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6938100b90 0x7f6938197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f693659c640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69380ff9a0 0x7f6938197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f6936d9d640 1 -- 172.21.15.38:0/1511585830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69380ff9a0 msgr2=0x7f6938197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f6936d9d640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69380ff9a0 0x7f6938197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f6936d9d640 1 -- 172.21.15.38:0/1511585830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693819f290 con 0x7f6938101550 2024-03-26T06:36:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f6936d9d640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6938101550 0x7f693819ec20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6930037040 tx=0x7f69300052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f692f7fe640 1 -- 172.21.15.38:0/1511585830 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6930042020 con 0x7f6938101550 2024-03-26T06:36:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f693cf09640 1 -- 172.21.15.38:0/1511585830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693819f520 con 0x7f6938101550 2024-03-26T06:36:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f692f7fe640 1 -- 172.21.15.38:0/1511585830 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6930004ad0 con 0x7f6938101550 2024-03-26T06:36:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f692f7fe640 1 -- 172.21.15.38:0/1511585830 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6930003970 con 0x7f6938101550 2024-03-26T06:36:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.101+0000 7f693cf09640 1 -- 172.21.15.38:0/1511585830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693819fa70 con 0x7f6938101550 2024-03-26T06:36:51.107 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.102+0000 7f693cf09640 1 -- 172.21.15.38:0/1511585830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693819fc70 con 0x7f6938101550 2024-03-26T06:36:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.103+0000 7f692f7fe640 1 -- 172.21.15.38:0/1511585830 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f6930045020 con 0x7f6938101550 2024-03-26T06:36:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.104+0000 7f692f7fe640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69240a2890 0x7f69240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.104+0000 7f693659c640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69240a2890 0x7f69240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.104+0000 7f692f7fe640 1 -- 172.21.15.38:0/1511585830 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f69300f1340 con 0x7f6938101550 2024-03-26T06:36:51.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.105+0000 7f693659c640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69240a2890 0x7f69240a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6928005c40 tx=0x7f6928005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:51.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.109+0000 7f692f7fe640 1 -- 172.21.15.38:0/1511585830 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f693004c020 con 0x7f6938101550 2024-03-26T06:36:51.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.291+0000 7f693cf09640 1 -- 172.21.15.38:0/1511585830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f69381a0000 con 0x7f6938101550 2024-03-26T06:36:51.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.292+0000 7f692f7fe640 1 -- 172.21.15.38:0/1511585830 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f69300be390 con 0x7f6938101550 2024-03-26T06:36:51.294 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-03-26T06:36:51.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 -- 172.21.15.38:0/1511585830 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69240a2890 msgr2=0x7f69240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69240a2890 0x7f69240a4d50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6928005c40 tx=0x7f6928005bd0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 -- 172.21.15.38:0/1511585830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6938101550 msgr2=0x7f693819ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6938101550 0x7f693819ec20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6930037040 tx=0x7f69300052b0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 -- 172.21.15.38:0/1511585830 shutdown_connections 2024-03-26T06:36:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f69240a2890 0x7f69240a4d50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6938101550 0x7f693819ec20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6938100b90 0x7f6938197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 --2- 172.21.15.38:0/1511585830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f69380ff9a0 0x7f6938197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 -- 172.21.15.38:0/1511585830 >> 172.21.15.38:0/1511585830 conn(0x7f69380faef0 msgr2=0x7f6938105fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 -- 172.21.15.38:0/1511585830 shutdown_connections 2024-03-26T06:36:51.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.294+0000 7f693cf09640 1 -- 172.21.15.38:0/1511585830 wait complete. 2024-03-26T06:36:51.307 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-26T06:36:51.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.410+0000 7fcb3c259640 1 -- 172.21.15.38:0/3244513311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34101da0 msgr2=0x7fcb34108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.410+0000 7fcb3c259640 1 --2- 172.21.15.38:0/3244513311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34101da0 0x7fcb34108630 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c009ca0 tx=0x7fcb2c02c0f0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 -- 172.21.15.38:0/3244513311 shutdown_connections 2024-03-26T06:36:51.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 --2- 172.21.15.38:0/3244513311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34101da0 0x7fcb34108630 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 --2- 172.21.15.38:0/3244513311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb341013e0 0x7fcb34101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 --2- 172.21.15.38:0/3244513311 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb341001f0 0x7fcb34100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 -- 172.21.15.38:0/3244513311 >> 172.21.15.38:0/3244513311 conn(0x7fcb340fb780 msgr2=0x7fcb340fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 -- 172.21.15.38:0/3244513311 shutdown_connections 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 -- 172.21.15.38:0/3244513311 wait complete. 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 Processor -- start 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 -- start start 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb341001f0 0x7fcb34068fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb341013e0 0x7fcb3419c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34101da0 0x7fcb3419ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3410bf60 con 0x7fcb34101da0 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3410bde0 con 0x7fcb341013e0 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3c259640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3410c0e0 con 0x7fcb341001f0 2024-03-26T06:36:51.416 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3a7f6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34101da0 0x7fcb3419ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.411+0000 7fcb3a7f6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34101da0 0x7fcb3419ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43194/0 (socket says 172.21.15.38:43194) 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb3a7f6640 1 -- 172.21.15.38:0/1507120684 learned_addr learned my addr 172.21.15.38:0/1507120684 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb39ff5640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb341001f0 0x7fcb34068fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb3a7f6640 1 -- 172.21.15.38:0/1507120684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb341001f0 msgr2=0x7fcb34068fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb3a7f6640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb341001f0 0x7fcb34068fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb3a7f6640 1 -- 172.21.15.38:0/1507120684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb341013e0 msgr2=0x7fcb3419c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb3a7f6640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb341013e0 0x7fcb3419c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb3a7f6640 1 -- 172.21.15.38:0/1507120684 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3419f300 con 0x7fcb34101da0 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb3a7f6640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34101da0 0x7fcb3419ea40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c037040 tx=0x7fcb2c032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb2affd640 1 -- 172.21.15.38:0/1507120684 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c039040 con 0x7fcb34101da0 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb2affd640 1 -- 172.21.15.38:0/1507120684 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb2c044070 con 0x7fcb34101da0 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb3c259640 1 -- 172.21.15.38:0/1507120684 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3419f590 con 0x7fcb34101da0 2024-03-26T06:36:51.417 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb2affd640 1 -- 172.21.15.38:0/1507120684 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c004330 con 0x7fcb34101da0 2024-03-26T06:36:51.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.412+0000 7fcb3c259640 1 -- 172.21.15.38:0/1507120684 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3419fb00 con 0x7fcb34101da0 2024-03-26T06:36:51.418 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.413+0000 7fcb3c259640 1 -- 172.21.15.38:0/1507120684 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb34107510 con 0x7fcb34101da0 2024-03-26T06:36:51.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.414+0000 7fcb2affd640 1 -- 172.21.15.38:0/1507120684 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fcb2c003a00 con 0x7fcb34101da0 2024-03-26T06:36:51.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.415+0000 7fcb2affd640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcb200a2890 0x7fcb200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.415+0000 7fcb39ff5640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcb200a2890 0x7fcb200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:51.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.415+0000 7fcb2affd640 1 -- 172.21.15.38:0/1507120684 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcb2c0f1400 con 0x7fcb34101da0 2024-03-26T06:36:51.419 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.415+0000 7fcb39ff5640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcb200a2890 0x7fcb200a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcb34101240 tx=0x7fcb30008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:51.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.419+0000 7fcb2affd640 1 -- 172.21.15.38:0/1507120684 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcb2c0be410 con 0x7fcb34101da0 2024-03-26T06:36:51.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:51 smithi038 ceph-mon[32265]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:51.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:51 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2897781542' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:36:51.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:51 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3996052268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:36:51.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:51 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1511585830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:36:51.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.604+0000 7fcb3c259640 1 -- 172.21.15.38:0/1507120684 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb3402d0c0 con 0x7fcb34101da0 2024-03-26T06:36:51.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.604+0000 7fcb2affd640 1 -- 172.21.15.38:0/1507120684 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb2c0bdd10 con 0x7fcb34101da0 2024-03-26T06:36:51.607 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-03-26T06:36:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 -- 172.21.15.38:0/1507120684 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcb200a2890 msgr2=0x7fcb200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcb200a2890 0x7fcb200a4d50 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcb34101240 tx=0x7fcb30008040 comp rx=0 tx=0).stop 2024-03-26T06:36:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 -- 172.21.15.38:0/1507120684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34101da0 msgr2=0x7fcb3419ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34101da0 0x7fcb3419ea40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c037040 tx=0x7fcb2c032f90 comp rx=0 tx=0).stop 2024-03-26T06:36:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 -- 172.21.15.38:0/1507120684 shutdown_connections 2024-03-26T06:36:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcb200a2890 0x7fcb200a4d50 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34101da0 0x7fcb3419ea40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb341013e0 0x7fcb3419c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 --2- 172.21.15.38:0/1507120684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb341001f0 0x7fcb34068fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 -- 172.21.15.38:0/1507120684 >> 172.21.15.38:0/1507120684 conn(0x7fcb340fb780 msgr2=0x7fcb341046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:51.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 -- 172.21.15.38:0/1507120684 shutdown_connections 2024-03-26T06:36:51.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.606+0000 7fcb3c259640 1 -- 172.21.15.38:0/1507120684 wait complete. 2024-03-26T06:36:51.620 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:36:51.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:51 smithi121 ceph-mon[36349]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:51.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:51 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2897781542' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:36:51.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:51 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3996052268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:36:51.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:51 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1511585830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:36:51.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:51 smithi170 ceph-mon[36912]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:36:51.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:51 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2897781542' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:36:51.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:51 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3996052268' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:36:51.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:51 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1511585830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:36:51.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.720+0000 7f5e16012640 1 -- 172.21.15.38:0/1761672428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e10104740 msgr2=0x7f5e10106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.720+0000 7f5e16012640 1 --2- 172.21.15.38:0/1761672428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e10104740 0x7f5e10106b30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5e04009900 tx=0x7f5e0402bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:51.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.720+0000 7f5e16012640 1 -- 172.21.15.38:0/1761672428 shutdown_connections 2024-03-26T06:36:51.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.720+0000 7f5e16012640 1 --2- 172.21.15.38:0/1761672428 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e10107070 0x7f5e101094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.720+0000 7f5e16012640 1 --2- 172.21.15.38:0/1761672428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e10104740 0x7f5e10106b30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.720+0000 7f5e16012640 1 --2- 172.21.15.38:0/1761672428 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e10101e10 0x7f5e10104200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.720+0000 7f5e16012640 1 -- 172.21.15.38:0/1761672428 >> 172.21.15.38:0/1761672428 conn(0x7f5e100fb820 msgr2=0x7f5e100fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:51.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.720+0000 7f5e16012640 1 -- 172.21.15.38:0/1761672428 shutdown_connections 2024-03-26T06:36:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.720+0000 7f5e16012640 1 -- 172.21.15.38:0/1761672428 wait complete. 2024-03-26T06:36:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.720+0000 7f5e16012640 1 Processor -- start 2024-03-26T06:36:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e16012640 1 -- start start 2024-03-26T06:36:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e16012640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e10101e10 0x7f5e10197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e16012640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e10104740 0x7f5e10197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e16012640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e10107070 0x7f5e1019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e16012640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1010cfb0 con 0x7f5e10107070 2024-03-26T06:36:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e16012640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1010ce30 con 0x7f5e10104740 2024-03-26T06:36:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e16012640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1010d130 con 0x7f5e10101e10 2024-03-26T06:36:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e10107070 0x7f5e1019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e10107070 0x7f5e1019ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43212/0 (socket says 172.21.15.38:43212) 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0ffff640 1 -- 172.21.15.38:0/583705869 learned_addr learned my addr 172.21.15.38:0/583705869 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0effd640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e10104740 0x7f5e10197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0f7fe640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e10101e10 0x7f5e10197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0ffff640 1 -- 172.21.15.38:0/583705869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e10101e10 msgr2=0x7f5e10197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0ffff640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e10101e10 0x7f5e10197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0ffff640 1 -- 172.21.15.38:0/583705869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e10104740 msgr2=0x7f5e10197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0ffff640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e10104740 0x7f5e10197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0ffff640 1 -- 172.21.15.38:0/583705869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1019f290 con 0x7f5e10107070 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0f7fe640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e10101e10 0x7f5e10197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0ffff640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e10107070 0x7f5e1019ec20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009fd0 tx=0x7f5e0800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.721+0000 7f5e0cff9640 1 -- 172.21.15.38:0/583705869 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e08017070 con 0x7f5e10107070 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.722+0000 7f5e0cff9640 1 -- 172.21.15.38:0/583705869 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e08003e50 con 0x7f5e10107070 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.722+0000 7f5e16012640 1 -- 172.21.15.38:0/583705869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1019f580 con 0x7f5e10107070 2024-03-26T06:36:51.727 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.722+0000 7f5e0cff9640 1 -- 172.21.15.38:0/583705869 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e080106c0 con 0x7f5e10107070 2024-03-26T06:36:51.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.722+0000 7f5e16012640 1 -- 172.21.15.38:0/583705869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1019fa50 con 0x7f5e10107070 2024-03-26T06:36:51.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.723+0000 7f5e16012640 1 -- 172.21.15.38:0/583705869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e100fd310 con 0x7f5e10107070 2024-03-26T06:36:51.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.724+0000 7f5e0cff9640 1 -- 172.21.15.38:0/583705869 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f5e08003970 con 0x7f5e10107070 2024-03-26T06:36:51.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.724+0000 7f5e0cff9640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5df80a2890 0x7f5df80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:51.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.724+0000 7f5e0f7fe640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5df80a2890 0x7f5df80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:51.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.724+0000 7f5e0cff9640 1 -- 172.21.15.38:0/583705869 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f5e080ce9a0 con 0x7f5e10107070 2024-03-26T06:36:51.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.725+0000 7f5e0f7fe640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5df80a2890 0x7f5df80a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009c20 tx=0x7f5e00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:51.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.729+0000 7f5e0cff9640 1 -- 172.21.15.38:0/583705869 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5e0809b8e0 con 0x7f5e10107070 2024-03-26T06:36:51.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.951+0000 7f5e16012640 1 -- 172.21.15.38:0/583705869 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e1002ce70 con 0x7f5e10107070 2024-03-26T06:36:51.954 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.952+0000 7f5e0cff9640 1 -- 172.21.15.38:0/583705869 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11346 (secure 0 0 0) 0x7f5e0809b1e0 con 0x7f5e10107070 2024-03-26T06:36:51.954 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:36:51.954 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/357411663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":357411663},{"type":"v1","addr":"172.21.15.38:6839","nonce":357411663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/654319512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":654319512},{"type":"v1","addr":"172.21.15.38:6841","nonce":654319512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-26T06:35:26.477444+0000","modified":"2024-03-26T06:35:32.167599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24502,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/359437185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":359437185},{"type":"v1","addr":"172.21.15.121:6839","nonce":359437185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-26T06:36:51.954 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 22 2024-03-26T06:36:51.956 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 -- 172.21.15.38:0/583705869 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5df80a2890 msgr2=0x7f5df80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5df80a2890 0x7f5df80a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5e00009c20 tx=0x7f5e00005f90 comp rx=0 tx=0).stop 2024-03-26T06:36:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 -- 172.21.15.38:0/583705869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e10107070 msgr2=0x7f5e1019ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e10107070 0x7f5e1019ec20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009fd0 tx=0x7f5e0800b260 comp rx=0 tx=0).stop 2024-03-26T06:36:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 -- 172.21.15.38:0/583705869 shutdown_connections 2024-03-26T06:36:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f5df80a2890 0x7f5df80a4d50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5e10107070 0x7f5e1019ec20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5e10104740 0x7f5e10197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 --2- 172.21.15.38:0/583705869 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5e10101e10 0x7f5e10197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 -- 172.21.15.38:0/583705869 >> 172.21.15.38:0/583705869 conn(0x7f5e100fb820 msgr2=0x7f5e10102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 -- 172.21.15.38:0/583705869 shutdown_connections 2024-03-26T06:36:51.957 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:51.954+0000 7f5e16012640 1 -- 172.21.15.38:0/583705869 wait complete. 2024-03-26T06:36:51.968 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:36:52.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.068+0000 7fdb18c59640 1 -- 172.21.15.38:0/3903962533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb14101480 msgr2=0x7fdb14101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.068+0000 7fdb18c59640 1 --2- 172.21.15.38:0/3903962533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb14101480 0x7fdb14101900 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdafc009900 tx=0x7fdafc02bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:52.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.068+0000 7fdb18c59640 1 -- 172.21.15.38:0/3903962533 shutdown_connections 2024-03-26T06:36:52.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.068+0000 7fdb18c59640 1 --2- 172.21.15.38:0/3903962533 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14101e40 0x7fdb141086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.068+0000 7fdb18c59640 1 --2- 172.21.15.38:0/3903962533 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb14101480 0x7fdb14101900 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.068+0000 7fdb18c59640 1 --2- 172.21.15.38:0/3903962533 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14100290 0x7fdb141006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.068+0000 7fdb18c59640 1 -- 172.21.15.38:0/3903962533 >> 172.21.15.38:0/3903962533 conn(0x7fdb140fb820 msgr2=0x7fdb140fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:52.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.068+0000 7fdb18c59640 1 -- 172.21.15.38:0/3903962533 shutdown_connections 2024-03-26T06:36:52.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.068+0000 7fdb18c59640 1 -- 172.21.15.38:0/3903962533 wait complete. 2024-03-26T06:36:52.073 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb18c59640 1 Processor -- start 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb18c59640 1 -- start start 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb18c59640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14100290 0x7fdb14104220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb18c59640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb14101480 0x7fdb14102860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb18c59640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14101e40 0x7fdb14102da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb18c59640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1410beb0 con 0x7fdb14101480 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb18c59640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1410bd30 con 0x7fdb14101e40 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb18c59640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1410c030 con 0x7fdb14100290 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb0bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb14101480 0x7fdb14102860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb0bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb14101480 0x7fdb14102860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43228/0 (socket says 172.21.15.38:43228) 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb0bfff640 1 -- 172.21.15.38:0/586573347 learned_addr learned my addr 172.21.15.38:0/586573347 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb12d9d640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14101e40 0x7fdb14102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.074 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb1259c640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14100290 0x7fdb14104220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb0bfff640 1 -- 172.21.15.38:0/586573347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14100290 msgr2=0x7fdb14104220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb0bfff640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14100290 0x7fdb14104220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb0bfff640 1 -- 172.21.15.38:0/586573347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14101e40 msgr2=0x7fdb14102da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb0bfff640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14101e40 0x7fdb14102da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb0bfff640 1 -- 172.21.15.38:0/586573347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb14103630 con 0x7fdb14101480 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb12d9d640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14101e40 0x7fdb14102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.069+0000 7fdb0bfff640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb14101480 0x7fdb14102860 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdafc009230 tx=0x7fdafc005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.070+0000 7fdb0b7fe640 1 -- 172.21.15.38:0/586573347 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc041070 con 0x7fdb14101480 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.070+0000 7fdb0b7fe640 1 -- 172.21.15.38:0/586573347 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdafc0054e0 con 0x7fdb14101480 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.070+0000 7fdb0b7fe640 1 -- 172.21.15.38:0/586573347 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc034440 con 0x7fdb14101480 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.070+0000 7fdb18c59640 1 -- 172.21.15.38:0/586573347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb141a16a0 con 0x7fdb14101480 2024-03-26T06:36:52.075 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.070+0000 7fdb18c59640 1 -- 172.21.15.38:0/586573347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb141a1ba0 con 0x7fdb14101480 2024-03-26T06:36:52.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.071+0000 7fdb18c59640 1 -- 172.21.15.38:0/586573347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb141a1e00 con 0x7fdb14101480 2024-03-26T06:36:52.076 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.072+0000 7fdb0b7fe640 1 -- 172.21.15.38:0/586573347 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fdafc044070 con 0x7fdb14101480 2024-03-26T06:36:52.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.073+0000 7fdb0b7fe640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb000a2910 0x7fdb000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.073+0000 7fdb1259c640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb000a2910 0x7fdb000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.073+0000 7fdb0b7fe640 1 -- 172.21.15.38:0/586573347 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdafc0f0820 con 0x7fdb14101480 2024-03-26T06:36:52.077 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.073+0000 7fdb1259c640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb000a2910 0x7fdb000a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdb04005fd0 tx=0x7fdb04005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:52.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.077+0000 7fdb0b7fe640 1 -- 172.21.15.38:0/586573347 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdafc0bd870 con 0x7fdb14101480 2024-03-26T06:36:52.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.262+0000 7fdb18c59640 1 -- 172.21.15.38:0/586573347 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb141a2190 con 0x7fdb14101480 2024-03-26T06:36:52.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.264+0000 7fdb0b7fe640 1 -- 172.21.15.38:0/586573347 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22634 (secure 0 0 0) 0x7fdafc0bd170 con 0x7fdb14101480 2024-03-26T06:36:52.267 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:36:52.267 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":88,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:36:27.512655+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:36:52.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 -- 172.21.15.38:0/586573347 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb000a2910 msgr2=0x7fdb000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb000a2910 0x7fdb000a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdb04005fd0 tx=0x7fdb04005850 comp rx=0 tx=0).stop 2024-03-26T06:36:52.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 -- 172.21.15.38:0/586573347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb14101480 msgr2=0x7fdb14102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb14101480 0x7fdb14102860 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdafc009230 tx=0x7fdafc005dd0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 -- 172.21.15.38:0/586573347 shutdown_connections 2024-03-26T06:36:52.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb000a2910 0x7fdb000a4dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb14101e40 0x7fdb14102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb14101480 0x7fdb14102860 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 --2- 172.21.15.38:0/586573347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb14100290 0x7fdb14104220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 -- 172.21.15.38:0/586573347 >> 172.21.15.38:0/586573347 conn(0x7fdb140fb820 msgr2=0x7fdb140fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:52.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 -- 172.21.15.38:0/586573347 shutdown_connections 2024-03-26T06:36:52.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.266+0000 7fdb18c59640 1 -- 172.21.15.38:0/586573347 wait complete. 2024-03-26T06:36:52.280 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T06:30:52.485897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-26T06:35:24.501259+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-26T06:35:25.675299+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T06:36:52.280 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T06:36:52.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.384+0000 7fb207d09640 1 -- 172.21.15.38:0/2633995648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb200101400 msgr2=0x7fb200101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.384+0000 7fb207d09640 1 --2- 172.21.15.38:0/2633995648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb200101400 0x7fb200101880 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009900 tx=0x7fb1f402bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:52.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.384+0000 7fb207d09640 1 -- 172.21.15.38:0/2633995648 shutdown_connections 2024-03-26T06:36:52.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.384+0000 7fb207d09640 1 --2- 172.21.15.38:0/2633995648 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb200101dc0 0x7fb200108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.384+0000 7fb207d09640 1 --2- 172.21.15.38:0/2633995648 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb200101400 0x7fb200101880 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.384+0000 7fb207d09640 1 --2- 172.21.15.38:0/2633995648 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb200100210 0x7fb200100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.384+0000 7fb207d09640 1 -- 172.21.15.38:0/2633995648 >> 172.21.15.38:0/2633995648 conn(0x7fb2000fb7a0 msgr2=0x7fb2000fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.384+0000 7fb207d09640 1 -- 172.21.15.38:0/2633995648 shutdown_connections 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.384+0000 7fb207d09640 1 -- 172.21.15.38:0/2633995648 wait complete. 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.384+0000 7fb207d09640 1 Processor -- start 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.384+0000 7fb207d09640 1 -- start start 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb207d09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb200100210 0x7fb2001041a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb207d09640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb200101400 0x7fb2001027e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb207d09640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb200101dc0 0x7fb200102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb207d09640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20010bea0 con 0x7fb200100210 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb207d09640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20010bd20 con 0x7fb200101400 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb207d09640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20010c020 con 0x7fb200101dc0 2024-03-26T06:36:52.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb205aa5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb200100210 0x7fb2001041a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb205aa5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb200100210 0x7fb2001041a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43248/0 (socket says 172.21.15.38:43248) 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb205aa5640 1 -- 172.21.15.38:0/1158278 learned_addr learned my addr 172.21.15.38:0/1158278 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb2052a4640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb200101400 0x7fb2001027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb2062a6640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb200101dc0 0x7fb200102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb205aa5640 1 -- 172.21.15.38:0/1158278 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb200101dc0 msgr2=0x7fb200102d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb205aa5640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb200101dc0 0x7fb200102d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb205aa5640 1 -- 172.21.15.38:0/1158278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb200101400 msgr2=0x7fb2001027e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb205aa5640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb200101400 0x7fb2001027e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb205aa5640 1 -- 172.21.15.38:0/1158278 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2001035b0 con 0x7fb200100210 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb205aa5640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb200100210 0x7fb2001041a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800e9c0 tx=0x7fb1f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb1feffd640 1 -- 172.21.15.38:0/1158278 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f800cd80 con 0x7fb200100210 2024-03-26T06:36:52.390 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb1feffd640 1 -- 172.21.15.38:0/1158278 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1f80035f0 con 0x7fb200100210 2024-03-26T06:36:52.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb207d09640 1 -- 172.21.15.38:0/1158278 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2001a1760 con 0x7fb200100210 2024-03-26T06:36:52.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb1feffd640 1 -- 172.21.15.38:0/1158278 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8010690 con 0x7fb200100210 2024-03-26T06:36:52.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.385+0000 7fb207d09640 1 -- 172.21.15.38:0/1158278 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2001a1c30 con 0x7fb200100210 2024-03-26T06:36:52.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.386+0000 7fb207d09640 1 -- 172.21.15.38:0/1158278 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb200068530 con 0x7fb200100210 2024-03-26T06:36:52.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.388+0000 7fb1feffd640 1 -- 172.21.15.38:0/1158278 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb1f80037a0 con 0x7fb200100210 2024-03-26T06:36:52.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.388+0000 7fb1feffd640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1e80a2890 0x7fb1e80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.388+0000 7fb2052a4640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1e80a2890 0x7fb1e80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.388+0000 7fb1feffd640 1 -- 172.21.15.38:0/1158278 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb1f80cf0b0 con 0x7fb200100210 2024-03-26T06:36:52.392 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.389+0000 7fb2052a4640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1e80a2890 0x7fb1e80a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb1f402c7b0 tx=0x7fb1f4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:52.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.392+0000 7fb1feffd640 1 -- 172.21.15.38:0/1158278 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb1f809bff0 con 0x7fb200100210 2024-03-26T06:36:52.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.573+0000 7fb207d09640 1 -- 172.21.15.38:0/1158278 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2001a2110 con 0x7fb200100210 2024-03-26T06:36:52.576 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.574+0000 7fb1feffd640 1 -- 172.21.15.38:0/1158278 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb1f809b8f0 con 0x7fb200100210 2024-03-26T06:36:52.577 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-03-26T06:36:52.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 -- 172.21.15.38:0/1158278 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1e80a2890 msgr2=0x7fb1e80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1e80a2890 0x7fb1e80a4d50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb1f402c7b0 tx=0x7fb1f4005fd0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 -- 172.21.15.38:0/1158278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb200100210 msgr2=0x7fb2001041a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb200100210 0x7fb2001041a0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800e9c0 tx=0x7fb1f800ee80 comp rx=0 tx=0).stop 2024-03-26T06:36:52.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 -- 172.21.15.38:0/1158278 shutdown_connections 2024-03-26T06:36:52.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb1e80a2890 0x7fb1e80a4d50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb200101dc0 0x7fb200102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb200101400 0x7fb2001027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 --2- 172.21.15.38:0/1158278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb200100210 0x7fb2001041a0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 -- 172.21.15.38:0/1158278 >> 172.21.15.38:0/1158278 conn(0x7fb2000fb7a0 msgr2=0x7fb2000fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:52.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 -- 172.21.15.38:0/1158278 shutdown_connections 2024-03-26T06:36:52.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.576+0000 7fb207d09640 1 -- 172.21.15.38:0/1158278 wait complete. 2024-03-26T06:36:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:52 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1507120684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:36:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:52 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/583705869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:36:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:52 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/586573347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:36:52.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-26T06:36:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:52 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1507120684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:36:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:52 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/583705869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:36:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:52 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/586573347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:36:52.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:52 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1507120684' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:36:52.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:52 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/583705869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:36:52.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:52 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/586573347' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:36:52.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 -- 172.21.15.38:0/1688726481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98580ff5f0 msgr2=0x7f98580ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 --2- 172.21.15.38:0/1688726481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98580ff5f0 0x7f98580ffa10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f984c009930 tx=0x7f984c02bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:52.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 -- 172.21.15.38:0/1688726481 shutdown_connections 2024-03-26T06:36:52.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 --2- 172.21.15.38:0/1688726481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102f80 0x7f9858109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 --2- 172.21.15.38:0/1688726481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98581025c0 0x7f9858102a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 --2- 172.21.15.38:0/1688726481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98580ff5f0 0x7f98580ffa10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 -- 172.21.15.38:0/1688726481 >> 172.21.15.38:0/1688726481 conn(0x7f98580faf20 msgr2=0x7f98580fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 -- 172.21.15.38:0/1688726481 shutdown_connections 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 -- 172.21.15.38:0/1688726481 wait complete. 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 Processor -- start 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 -- start start 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98580ff5f0 0x7f98581976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98581025c0 0x7f9858197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102f80 0x7f985819ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.693+0000 7f985ebc6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985810cfe0 con 0x7f98581025c0 2024-03-26T06:36:52.698 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f985ebc6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985810ce60 con 0x7f9858102f80 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f985ebc6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985810d160 con 0x7f98580ff5f0 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9857fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98581025c0 0x7f9858197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9857fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98581025c0 0x7f9858197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43262/0 (socket says 172.21.15.38:43262) 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9857fff640 1 -- 172.21.15.38:0/2714589016 learned_addr learned my addr 172.21.15.38:0/2714589016 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f985c962640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98580ff5f0 0x7f98581976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f985d163640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102f80 0x7f985819ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9857fff640 1 -- 172.21.15.38:0/2714589016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98580ff5f0 msgr2=0x7f98581976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9857fff640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98580ff5f0 0x7f98581976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9857fff640 1 -- 172.21.15.38:0/2714589016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102f80 msgr2=0x7f985819ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9857fff640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102f80 0x7f985819ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9857fff640 1 -- 172.21.15.38:0/2714589016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985819f2c0 con 0x7f98581025c0 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f985d163640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102f80 0x7f985819ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:52.699 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9857fff640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98581025c0 0x7f9858197be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f984400d770 tx=0x7f984400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:52.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9855ffb640 1 -- 172.21.15.38:0/2714589016 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844017070 con 0x7f98581025c0 2024-03-26T06:36:52.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9855ffb640 1 -- 172.21.15.38:0/2714589016 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9844005500 con 0x7f98581025c0 2024-03-26T06:36:52.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f985ebc6640 1 -- 172.21.15.38:0/2714589016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985819f5b0 con 0x7f98581025c0 2024-03-26T06:36:52.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.694+0000 7f9855ffb640 1 -- 172.21.15.38:0/2714589016 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9844004e00 con 0x7f98581025c0 2024-03-26T06:36:52.700 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.695+0000 7f985ebc6640 1 -- 172.21.15.38:0/2714589016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985819fab0 con 0x7f98581025c0 2024-03-26T06:36:52.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.696+0000 7f985ebc6640 1 -- 172.21.15.38:0/2714589016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9858108710 con 0x7f98581025c0 2024-03-26T06:36:52.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.697+0000 7f9855ffb640 1 -- 172.21.15.38:0/2714589016 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9844004fa0 con 0x7f98581025c0 2024-03-26T06:36:52.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.697+0000 7f9855ffb640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f98480a2890 0x7f98480a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:52.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.697+0000 7f985c962640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f98480a2890 0x7f98480a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:52.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.697+0000 7f9855ffb640 1 -- 172.21.15.38:0/2714589016 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f98440ce630 con 0x7f98581025c0 2024-03-26T06:36:52.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.698+0000 7f985c962640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f98480a2890 0x7f98480a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f984c02c7b0 tx=0x7f984c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:52.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.702+0000 7f9855ffb640 1 -- 172.21.15.38:0/2714589016 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f984409b570 con 0x7f98581025c0 2024-03-26T06:36:52.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.884+0000 7f985ebc6640 1 -- 172.21.15.38:0/2714589016 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f985802d0c0 con 0x7f98581025c0 2024-03-26T06:36:52.887 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.885+0000 7f9855ffb640 1 -- 172.21.15.38:0/2714589016 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f984409ae70 con 0x7f98581025c0 2024-03-26T06:36:52.888 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-03-26T06:36:52.889 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 -- 172.21.15.38:0/2714589016 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f98480a2890 msgr2=0x7f98480a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f98480a2890 0x7f98480a4d50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f984c02c7b0 tx=0x7f984c000bc0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 -- 172.21.15.38:0/2714589016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98581025c0 msgr2=0x7f9858197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98581025c0 0x7f9858197be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f984400d770 tx=0x7f984400dc30 comp rx=0 tx=0).stop 2024-03-26T06:36:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 -- 172.21.15.38:0/2714589016 shutdown_connections 2024-03-26T06:36:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f98480a2890 0x7f98480a4d50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9858102f80 0x7f985819ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f98581025c0 0x7f9858197be0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 --2- 172.21.15.38:0/2714589016 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f98580ff5f0 0x7f98581976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 -- 172.21.15.38:0/2714589016 >> 172.21.15.38:0/2714589016 conn(0x7f98580faf20 msgr2=0x7f98581058e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 -- 172.21.15.38:0/2714589016 shutdown_connections 2024-03-26T06:36:52.890 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:52.887+0000 7f985ebc6640 1 -- 172.21.15.38:0/2714589016 wait complete. 2024-03-26T06:36:52.900 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-26T06:36:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.003+0000 7f19e88d7640 1 -- 172.21.15.38:0/284586049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19e41025a0 msgr2=0x7f19e4102a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.003+0000 7f19e88d7640 1 --2- 172.21.15.38:0/284586049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19e41025a0 0x7f19e4102a20 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f19cc009930 tx=0x7f19cc02bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.003+0000 7f19e88d7640 1 -- 172.21.15.38:0/284586049 shutdown_connections 2024-03-26T06:36:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.003+0000 7f19e88d7640 1 --2- 172.21.15.38:0/284586049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e4102f60 0x7f19e41097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.003+0000 7f19e88d7640 1 --2- 172.21.15.38:0/284586049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19e41025a0 0x7f19e4102a20 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.003+0000 7f19e88d7640 1 --2- 172.21.15.38:0/284586049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19e40ff5d0 0x7f19e40ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.003+0000 7f19e88d7640 1 -- 172.21.15.38:0/284586049 >> 172.21.15.38:0/284586049 conn(0x7f19e40faf00 msgr2=0x7f19e40fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.003+0000 7f19e88d7640 1 -- 172.21.15.38:0/284586049 shutdown_connections 2024-03-26T06:36:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.003+0000 7f19e88d7640 1 -- 172.21.15.38:0/284586049 wait complete. 2024-03-26T06:36:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e88d7640 1 Processor -- start 2024-03-26T06:36:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e88d7640 1 -- start start 2024-03-26T06:36:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e88d7640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19e40ff5d0 0x7f19e4197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e88d7640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19e41025a0 0x7f19e4197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e88d7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e4102f60 0x7f19e419ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e88d7640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e410cfc0 con 0x7f19e40ff5d0 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e88d7640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e410ce40 con 0x7f19e4102f60 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e88d7640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e410d140 con 0x7f19e41025a0 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19e40ff5d0 0x7f19e4197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19e40ff5d0 0x7f19e4197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43282/0 (socket says 172.21.15.38:43282) 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e259c640 1 -- 172.21.15.38:0/3106138856 learned_addr learned my addr 172.21.15.38:0/3106138856 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e2d9d640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e4102f60 0x7f19e419ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19dbfff640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19e41025a0 0x7f19e4197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e259c640 1 -- 172.21.15.38:0/3106138856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19e41025a0 msgr2=0x7f19e4197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e259c640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19e41025a0 0x7f19e4197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e259c640 1 -- 172.21.15.38:0/3106138856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e4102f60 msgr2=0x7f19e419ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e259c640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e4102f60 0x7f19e419ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e259c640 1 -- 172.21.15.38:0/3106138856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e419f270 con 0x7f19e40ff5d0 2024-03-26T06:36:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e2d9d640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e4102f60 0x7f19e419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:36:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.004+0000 7f19e259c640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19e40ff5d0 0x7f19e4197650 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f19d400d950 tx=0x7f19d400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.005+0000 7f19db7fe640 1 -- 172.21.15.38:0/3106138856 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d4017070 con 0x7f19e40ff5d0 2024-03-26T06:36:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.005+0000 7f19e88d7640 1 -- 172.21.15.38:0/3106138856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e419f560 con 0x7f19e40ff5d0 2024-03-26T06:36:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.005+0000 7f19db7fe640 1 -- 172.21.15.38:0/3106138856 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19d4009c30 con 0x7f19e40ff5d0 2024-03-26T06:36:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.005+0000 7f19db7fe640 1 -- 172.21.15.38:0/3106138856 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d400f600 con 0x7f19e40ff5d0 2024-03-26T06:36:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.005+0000 7f19e88d7640 1 -- 172.21.15.38:0/3106138856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e419fa60 con 0x7f19e40ff5d0 2024-03-26T06:36:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.006+0000 7f19e88d7640 1 -- 172.21.15.38:0/3106138856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e4108670 con 0x7f19e40ff5d0 2024-03-26T06:36:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.007+0000 7f19db7fe640 1 -- 172.21.15.38:0/3106138856 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f19d401a070 con 0x7f19e40ff5d0 2024-03-26T06:36:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.007+0000 7f19db7fe640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f19d00a2890 0x7f19d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.007+0000 7f19dbfff640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f19d00a2890 0x7f19d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.007+0000 7f19db7fe640 1 -- 172.21.15.38:0/3106138856 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f19d40ce730 con 0x7f19e40ff5d0 2024-03-26T06:36:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.008+0000 7f19dbfff640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f19d00a2890 0x7f19d00a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f19cc036040 tx=0x7f19cc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.011+0000 7f19db7fe640 1 -- 172.21.15.38:0/3106138856 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f19d4009de0 con 0x7f19e40ff5d0 2024-03-26T06:36:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.192+0000 7f19e88d7640 1 -- 172.21.15.38:0/3106138856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f19e4061ef0 con 0x7f19e40ff5d0 2024-03-26T06:36:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.192+0000 7f19db7fe640 1 -- 172.21.15.38:0/3106138856 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f19d409b670 con 0x7f19e40ff5d0 2024-03-26T06:36:53.195 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-03-26T06:36:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.194+0000 7f19e88d7640 1 -- 172.21.15.38:0/3106138856 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f19d00a2890 msgr2=0x7f19d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.194+0000 7f19e88d7640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f19d00a2890 0x7f19d00a4d50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f19cc036040 tx=0x7f19cc005fd0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.194+0000 7f19e88d7640 1 -- 172.21.15.38:0/3106138856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19e40ff5d0 msgr2=0x7f19e4197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.194+0000 7f19e88d7640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19e40ff5d0 0x7f19e4197650 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f19d400d950 tx=0x7f19d400de10 comp rx=0 tx=0).stop 2024-03-26T06:36:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.194+0000 7f19e88d7640 1 -- 172.21.15.38:0/3106138856 shutdown_connections 2024-03-26T06:36:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.194+0000 7f19e88d7640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f19d00a2890 0x7f19d00a4d50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.194+0000 7f19e88d7640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19e4102f60 0x7f19e419ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.194+0000 7f19e88d7640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f19e41025a0 0x7f19e4197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.194+0000 7f19e88d7640 1 --2- 172.21.15.38:0/3106138856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19e40ff5d0 0x7f19e4197650 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.195+0000 7f19e88d7640 1 -- 172.21.15.38:0/3106138856 >> 172.21.15.38:0/3106138856 conn(0x7f19e40faf00 msgr2=0x7f19e4105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.195+0000 7f19e88d7640 1 -- 172.21.15.38:0/3106138856 shutdown_connections 2024-03-26T06:36:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.195+0000 7f19e88d7640 1 -- 172.21.15.38:0/3106138856 wait complete. 2024-03-26T06:36:53.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:36:53.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.308+0000 7f4dc851b640 1 -- 172.21.15.38:0/3369222020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dc0100210 msgr2=0x7f4dc0100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.308+0000 7f4dc851b640 1 --2- 172.21.15.38:0/3369222020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dc0100210 0x7f4dc0100630 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009900 tx=0x7f4dbc02bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:53.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.308+0000 7f4dc851b640 1 -- 172.21.15.38:0/3369222020 shutdown_connections 2024-03-26T06:36:53.313 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.308+0000 7f4dc851b640 1 --2- 172.21.15.38:0/3369222020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc0101dc0 0x7f4dc0108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.308+0000 7f4dc851b640 1 --2- 172.21.15.38:0/3369222020 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dc0101400 0x7f4dc0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.308+0000 7f4dc851b640 1 --2- 172.21.15.38:0/3369222020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dc0100210 0x7f4dc0100630 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.308+0000 7f4dc851b640 1 -- 172.21.15.38:0/3369222020 >> 172.21.15.38:0/3369222020 conn(0x7f4dc00fb780 msgr2=0x7f4dc00fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.308+0000 7f4dc851b640 1 -- 172.21.15.38:0/3369222020 shutdown_connections 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.308+0000 7f4dc851b640 1 -- 172.21.15.38:0/3369222020 wait complete. 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc851b640 1 Processor -- start 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc851b640 1 -- start start 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc851b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc0100210 0x7f4dc01975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc851b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dc0101400 0x7f4dc0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc851b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dc0101dc0 0x7f4dc019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc851b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc010bea0 con 0x7f4dc0101400 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc851b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc010bd20 con 0x7f4dc0100210 2024-03-26T06:36:53.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc851b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dc010c020 con 0x7f4dc0101dc0 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc5ab6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dc0101400 0x7f4dc0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc5ab6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dc0101400 0x7f4dc0197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43298/0 (socket says 172.21.15.38:43298) 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc5ab6640 1 -- 172.21.15.38:0/1459136603 learned_addr learned my addr 172.21.15.38:0/1459136603 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.309+0000 7f4dc62b7640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc0100210 0x7f4dc01975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.310+0000 7f4dc5ab6640 1 -- 172.21.15.38:0/1459136603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dc0101dc0 msgr2=0x7f4dc019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.310+0000 7f4dc5ab6640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dc0101dc0 0x7f4dc019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.310+0000 7f4dc5ab6640 1 -- 172.21.15.38:0/1459136603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc0100210 msgr2=0x7f4dc01975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.310+0000 7f4dc5ab6640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc0100210 0x7f4dc01975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.310+0000 7f4dc5ab6640 1 -- 172.21.15.38:0/1459136603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc019f1f0 con 0x7f4dc0101400 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.310+0000 7f4dc5ab6640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dc0101400 0x7f4dc0197b10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4db000c920 tx=0x7f4db000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.310+0000 7f4db77fe640 1 -- 172.21.15.38:0/1459136603 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0007c70 con 0x7f4dc0101400 2024-03-26T06:36:53.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.310+0000 7f4db77fe640 1 -- 172.21.15.38:0/1459136603 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4db0007e10 con 0x7f4dc0101400 2024-03-26T06:36:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.310+0000 7f4dc851b640 1 -- 172.21.15.38:0/1459136603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc019f4e0 con 0x7f4dc0101400 2024-03-26T06:36:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.310+0000 7f4db77fe640 1 -- 172.21.15.38:0/1459136603 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db0018890 con 0x7f4dc0101400 2024-03-26T06:36:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.310+0000 7f4dc851b640 1 -- 172.21.15.38:0/1459136603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dc019fa80 con 0x7f4dc0101400 2024-03-26T06:36:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.311+0000 7f4dc851b640 1 -- 172.21.15.38:0/1459136603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dc019fd40 con 0x7f4dc0101400 2024-03-26T06:36:53.316 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.312+0000 7f4db77fe640 1 -- 172.21.15.38:0/1459136603 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f4db0018a70 con 0x7f4dc0101400 2024-03-26T06:36:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.312+0000 7f4db77fe640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4dac0a2910 0x7f4dac0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.313+0000 7f4dc62b7640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4dac0a2910 0x7f4dac0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.313+0000 7f4db77fe640 1 -- 172.21.15.38:0/1459136603 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f4db00ced70 con 0x7f4dc0101400 2024-03-26T06:36:53.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.313+0000 7f4dc62b7640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4dac0a2910 0x7f4dac0a4dd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc02c7b0 tx=0x7f4dbc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:53.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.317+0000 7f4db77fe640 1 -- 172.21.15.38:0/1459136603 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4db0014030 con 0x7f4dc0101400 2024-03-26T06:36:53.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.538+0000 7f4dc851b640 1 -- 172.21.15.38:0/1459136603 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4dc019ffb0 con 0x7f4dc0101400 2024-03-26T06:36:53.541 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.539+0000 7f4db77fe640 1 -- 172.21.15.38:0/1459136603 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11346 (secure 0 0 0) 0x7f4db009bcb0 con 0x7f4dc0101400 2024-03-26T06:36:53.541 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:36:53.542 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/357411663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":357411663},{"type":"v1","addr":"172.21.15.38:6839","nonce":357411663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/654319512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":654319512},{"type":"v1","addr":"172.21.15.38:6841","nonce":654319512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-26T06:35:26.477444+0000","modified":"2024-03-26T06:35:32.167599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24502,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/359437185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":359437185},{"type":"v1","addr":"172.21.15.121:6839","nonce":359437185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-26T06:36:53.542 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 22 2024-03-26T06:36:53.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 -- 172.21.15.38:0/1459136603 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4dac0a2910 msgr2=0x7f4dac0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4dac0a2910 0x7f4dac0a4dd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc02c7b0 tx=0x7f4dbc005fd0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.543 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 -- 172.21.15.38:0/1459136603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dc0101400 msgr2=0x7f4dc0197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dc0101400 0x7f4dc0197b10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4db000c920 tx=0x7f4db000cde0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 -- 172.21.15.38:0/1459136603 shutdown_connections 2024-03-26T06:36:53.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f4dac0a2910 0x7f4dac0a4dd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4dc0101dc0 0x7f4dc019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4dc0101400 0x7f4dc0197b10 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 --2- 172.21.15.38:0/1459136603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4dc0100210 0x7f4dc01975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 -- 172.21.15.38:0/1459136603 >> 172.21.15.38:0/1459136603 conn(0x7f4dc00fb780 msgr2=0x7f4dc00fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:53.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 -- 172.21.15.38:0/1459136603 shutdown_connections 2024-03-26T06:36:53.544 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.541+0000 7f4dc851b640 1 -- 172.21.15.38:0/1459136603 wait complete. 2024-03-26T06:36:53.553 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:53 smithi038 ceph-mon[32265]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:53.553 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:53 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1158278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:36:53.553 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:53 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2714589016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:36:53.553 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:53 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3106138856' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:36:53.555 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:36:53.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:53 smithi121 ceph-mon[36349]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:53.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:53 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1158278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:36:53.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:53 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2714589016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:36:53.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:53 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3106138856' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.654+0000 7fae0be06640 1 -- 172.21.15.38:0/3360108751 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae041047e0 msgr2=0x7fae04106bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.654+0000 7fae0be06640 1 --2- 172.21.15.38:0/3360108751 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae041047e0 0x7fae04106bd0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fadf4009900 tx=0x7fadf402bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.654+0000 7fae0be06640 1 -- 172.21.15.38:0/3360108751 shutdown_connections 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.654+0000 7fae0be06640 1 --2- 172.21.15.38:0/3360108751 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae04107110 0x7fae04109500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.654+0000 7fae0be06640 1 --2- 172.21.15.38:0/3360108751 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae041047e0 0x7fae04106bd0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.654+0000 7fae0be06640 1 --2- 172.21.15.38:0/3360108751 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae04101eb0 0x7fae041042a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.654+0000 7fae0be06640 1 -- 172.21.15.38:0/3360108751 >> 172.21.15.38:0/3360108751 conn(0x7fae040fb820 msgr2=0x7fae040fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.654+0000 7fae0be06640 1 -- 172.21.15.38:0/3360108751 shutdown_connections 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.654+0000 7fae0be06640 1 -- 172.21.15.38:0/3360108751 wait complete. 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.654+0000 7fae0be06640 1 Processor -- start 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae0be06640 1 -- start start 2024-03-26T06:36:53.659 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae0be06640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae04101eb0 0x7fae041976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae0be06640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae041047e0 0x7fae04197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae0be06640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae04107110 0x7fae0419ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae0be06640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0410cff0 con 0x7fae041047e0 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae0be06640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0410ce70 con 0x7fae04107110 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae0be06640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0410d170 con 0x7fae04101eb0 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae093a1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae041047e0 0x7fae04197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae093a1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae041047e0 0x7fae04197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43316/0 (socket says 172.21.15.38:43316) 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae093a1640 1 -- 172.21.15.38:0/258703971 learned_addr learned my addr 172.21.15.38:0/258703971 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae0a3a3640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae04107110 0x7fae0419ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae09ba2640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae04101eb0 0x7fae041976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae093a1640 1 -- 172.21.15.38:0/258703971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae04101eb0 msgr2=0x7fae041976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae093a1640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae04101eb0 0x7fae041976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae093a1640 1 -- 172.21.15.38:0/258703971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae04107110 msgr2=0x7fae0419ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.660 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae093a1640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae04107110 0x7fae0419ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae093a1640 1 -- 172.21.15.38:0/258703971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0419f2c0 con 0x7fae041047e0 2024-03-26T06:36:53.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae093a1640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae041047e0 0x7fae04197be0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fadf4005ab0 tx=0x7fadf4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:53.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fadfaffd640 1 -- 172.21.15.38:0/258703971 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf4041070 con 0x7fae041047e0 2024-03-26T06:36:53.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fadfaffd640 1 -- 172.21.15.38:0/258703971 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fadf40054e0 con 0x7fae041047e0 2024-03-26T06:36:53.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fae0be06640 1 -- 172.21.15.38:0/258703971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0419f550 con 0x7fae041047e0 2024-03-26T06:36:53.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.655+0000 7fadfaffd640 1 -- 172.21.15.38:0/258703971 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadf4034440 con 0x7fae041047e0 2024-03-26T06:36:53.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.656+0000 7fae0be06640 1 -- 172.21.15.38:0/258703971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0419fa90 con 0x7fae041047e0 2024-03-26T06:36:53.661 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.656+0000 7fae0be06640 1 -- 172.21.15.38:0/258703971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae040fd310 con 0x7fae041047e0 2024-03-26T06:36:53.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.658+0000 7fadfaffd640 1 -- 172.21.15.38:0/258703971 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fadf4044070 con 0x7fae041047e0 2024-03-26T06:36:53.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.658+0000 7fadfaffd640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fadf00a2910 0x7fadf00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.658+0000 7fae09ba2640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fadf00a2910 0x7fadf00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.658+0000 7fadfaffd640 1 -- 172.21.15.38:0/258703971 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fadf40f09b0 con 0x7fae041047e0 2024-03-26T06:36:53.662 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.659+0000 7fae09ba2640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fadf00a2910 0x7fadf00a4dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fae00005fd0 tx=0x7fae00005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:53.664 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.662+0000 7fadfaffd640 1 -- 172.21.15.38:0/258703971 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fadf40bda00 con 0x7fae041047e0 2024-03-26T06:36:53.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:53 smithi170 ceph-mon[36912]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:36:53.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:53 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1158278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:36:53.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:53 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2714589016' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:36:53.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:53 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3106138856' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:36:53.850 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.848+0000 7fae0be06640 1 -- 172.21.15.38:0/258703971 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae0402d040 con 0x7fae041047e0 2024-03-26T06:36:53.851 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.849+0000 7fadfaffd640 1 -- 172.21.15.38:0/258703971 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22634 (secure 0 0 0) 0x7fadf40bd300 con 0x7fae041047e0 2024-03-26T06:36:53.852 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:36:53.852 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":88,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:36:27.512655+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:36:53.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 -- 172.21.15.38:0/258703971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fadf00a2910 msgr2=0x7fadf00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fadf00a2910 0x7fadf00a4dd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fae00005fd0 tx=0x7fae00005850 comp rx=0 tx=0).stop 2024-03-26T06:36:53.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 -- 172.21.15.38:0/258703971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae041047e0 msgr2=0x7fae04197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae041047e0 0x7fae04197be0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fadf4005ab0 tx=0x7fadf4005dc0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 -- 172.21.15.38:0/258703971 shutdown_connections 2024-03-26T06:36:53.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fadf00a2910 0x7fadf00a4dd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae04107110 0x7fae0419ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.854 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae041047e0 0x7fae04197be0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 --2- 172.21.15.38:0/258703971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fae04101eb0 0x7fae041976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 -- 172.21.15.38:0/258703971 >> 172.21.15.38:0/258703971 conn(0x7fae040fb820 msgr2=0x7fae04102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:53.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 -- 172.21.15.38:0/258703971 shutdown_connections 2024-03-26T06:36:53.855 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.851+0000 7fae0be06640 1 -- 172.21.15.38:0/258703971 wait complete. 2024-03-26T06:36:53.865 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:36:53.970 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.966+0000 7f1cb6643640 1 -- 172.21.15.38:0/3063677372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cb0101e30 msgr2=0x7f1cb0104220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.966+0000 7f1cb6643640 1 --2- 172.21.15.38:0/3063677372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cb0101e30 0x7f1cb0104220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1c98009900 tx=0x7f1c9802bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.966+0000 7f1cb6643640 1 -- 172.21.15.38:0/3063677372 shutdown_connections 2024-03-26T06:36:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.966+0000 7f1cb6643640 1 --2- 172.21.15.38:0/3063677372 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb0107090 0x7f1cb0109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.966+0000 7f1cb6643640 1 --2- 172.21.15.38:0/3063677372 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cb0104760 0x7f1cb0106b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.966+0000 7f1cb6643640 1 --2- 172.21.15.38:0/3063677372 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cb0101e30 0x7f1cb0104220 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.966+0000 7f1cb6643640 1 -- 172.21.15.38:0/3063677372 >> 172.21.15.38:0/3063677372 conn(0x7f1cb00fb780 msgr2=0x7f1cb00fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.966+0000 7f1cb6643640 1 -- 172.21.15.38:0/3063677372 shutdown_connections 2024-03-26T06:36:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.966+0000 7f1cb6643640 1 -- 172.21.15.38:0/3063677372 wait complete. 2024-03-26T06:36:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.966+0000 7f1cb6643640 1 Processor -- start 2024-03-26T06:36:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.966+0000 7f1cb6643640 1 -- start start 2024-03-26T06:36:53.971 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cb6643640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb0101e30 0x7f1cb01975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cb6643640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cb0104760 0x7f1cb0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cb6643640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cb0107090 0x7f1cb019eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cb6643640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb010cfc0 con 0x7f1cb0104760 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cb6643640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb010ce40 con 0x7f1cb0101e30 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cb6643640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb010d140 con 0x7f1cb0107090 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1caf7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cb0104760 0x7f1cb0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1caf7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cb0104760 0x7f1cb0197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43328/0 (socket says 172.21.15.38:43328) 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1caf7fe640 1 -- 172.21.15.38:0/3183548765 learned_addr learned my addr 172.21.15.38:0/3183548765 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cb4be0640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cb0107090 0x7f1cb019eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1caffff640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb0101e30 0x7f1cb01975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1caf7fe640 1 -- 172.21.15.38:0/3183548765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cb0107090 msgr2=0x7f1cb019eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1caf7fe640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cb0107090 0x7f1cb019eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1caf7fe640 1 -- 172.21.15.38:0/3183548765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb0101e30 msgr2=0x7f1cb01975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:53.972 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1caf7fe640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb0101e30 0x7f1cb01975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1caf7fe640 1 -- 172.21.15.38:0/3183548765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb019f1f0 con 0x7f1cb0104760 2024-03-26T06:36:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1caffff640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb0101e30 0x7f1cb01975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1caf7fe640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cb0104760 0x7f1cb0197b10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000c970 tx=0x7f1ca000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cad7fa640 1 -- 172.21.15.38:0/3183548765 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0014070 con 0x7f1cb0104760 2024-03-26T06:36:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cad7fa640 1 -- 172.21.15.38:0/3183548765 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ca0004440 con 0x7f1cb0104760 2024-03-26T06:36:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cb6643640 1 -- 172.21.15.38:0/3183548765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb019f4e0 con 0x7f1cb0104760 2024-03-26T06:36:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cad7fa640 1 -- 172.21.15.38:0/3183548765 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca000f450 con 0x7f1cb0104760 2024-03-26T06:36:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.967+0000 7f1cb6643640 1 -- 172.21.15.38:0/3183548765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb019fa80 con 0x7f1cb0104760 2024-03-26T06:36:53.973 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.968+0000 7f1cb6643640 1 -- 172.21.15.38:0/3183548765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb00fd270 con 0x7f1cb0104760 2024-03-26T06:36:53.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.969+0000 7f1cad7fa640 1 -- 172.21.15.38:0/3183548765 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f1ca0017070 con 0x7f1cb0104760 2024-03-26T06:36:53.974 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.970+0000 7f1cad7fa640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c9c0a2910 0x7f1c9c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.970+0000 7f1caffff640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c9c0a2910 0x7f1c9c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.970+0000 7f1cad7fa640 1 -- 172.21.15.38:0/3183548765 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1ca00cead0 con 0x7f1cb0104760 2024-03-26T06:36:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.970+0000 7f1caffff640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c9c0a2910 0x7f1c9c0a4dd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1c980096c0 tx=0x7f1c98000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:53.978 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:53.974+0000 7f1cad7fa640 1 -- 172.21.15.38:0/3183548765 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1ca001e020 con 0x7f1cb0104760 2024-03-26T06:36:54.198 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.196+0000 7f1cb6643640 1 -- 172.21.15.38:0/3183548765 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cb002ce70 con 0x7f1cb0104760 2024-03-26T06:36:54.199 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.196+0000 7f1cad7fa640 1 -- 172.21.15.38:0/3183548765 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11346 (secure 0 0 0) 0x7f1ca009b9d0 con 0x7f1cb0104760 2024-03-26T06:36:54.199 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:36:54.199 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/357411663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":357411663},{"type":"v1","addr":"172.21.15.38:6839","nonce":357411663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/654319512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":654319512},{"type":"v1","addr":"172.21.15.38:6841","nonce":654319512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-26T06:35:26.477444+0000","modified":"2024-03-26T06:35:32.167599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24502,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/359437185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":359437185},{"type":"v1","addr":"172.21.15.121:6839","nonce":359437185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-26T06:36:54.199 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 22 2024-03-26T06:36:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 -- 172.21.15.38:0/3183548765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c9c0a2910 msgr2=0x7f1c9c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c9c0a2910 0x7f1c9c0a4dd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1c980096c0 tx=0x7f1c98000bc0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 -- 172.21.15.38:0/3183548765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cb0104760 msgr2=0x7f1cb0197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cb0104760 0x7f1cb0197b10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1ca000c970 tx=0x7f1ca000ce30 comp rx=0 tx=0).stop 2024-03-26T06:36:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 -- 172.21.15.38:0/3183548765 shutdown_connections 2024-03-26T06:36:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c9c0a2910 0x7f1c9c0a4dd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1cb0107090 0x7f1cb019eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cb0104760 0x7f1cb0197b10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 --2- 172.21.15.38:0/3183548765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cb0101e30 0x7f1cb01975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 -- 172.21.15.38:0/3183548765 >> 172.21.15.38:0/3183548765 conn(0x7f1cb00fb780 msgr2=0x7f1cb0102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 -- 172.21.15.38:0/3183548765 shutdown_connections 2024-03-26T06:36:54.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.198+0000 7f1cb6643640 1 -- 172.21.15.38:0/3183548765 wait complete. 2024-03-26T06:36:54.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-26T06:35:26.477444+0000', 'modified': '2024-03-26T06:35:32.167599+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24539, 'mds_1': 24502, 'mds_2': 24488}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6835/2374885191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6834', 'nonce': 2374885191}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 2374885191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24464': {'gid': 24464, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.121:6835/3384748966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6834', 'nonce': 3384748966}, {'type': 'v1', 'addr': '172.21.15.121:6835', 'nonce': 3384748966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6837/2484220351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6836', 'nonce': 2484220351}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 2484220351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.121:6837/1114423776', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6836', 'nonce': 1114423776}, {'type': 'v1', 'addr': '172.21.15.121:6837', 'nonce': 1114423776}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24502': {'gid': 24502, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.170:6839/2204378068', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6838', 'nonce': 2204378068}, {'type': 'v1', 'addr': '172.21.15.170:6839', 'nonce': 2204378068}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24539': {'gid': 24539, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6839/359437185', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6838', 'nonce': 359437185}, {'type': 'v1', 'addr': '172.21.15.121:6839', 'nonce': 359437185}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-26T06:36:54.212 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-26T06:36:54.213 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-26T06:36:54.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.314+0000 7f64ae5a9640 1 -- 172.21.15.38:0/2635171659 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a81024f0 msgr2=0x7f64a8102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.314+0000 7f64ae5a9640 1 --2- 172.21.15.38:0/2635171659 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a81024f0 0x7f64a8102970 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f649c009900 tx=0x7f649c02bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.314+0000 7f64ae5a9640 1 -- 172.21.15.38:0/2635171659 shutdown_connections 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.314+0000 7f64ae5a9640 1 --2- 172.21.15.38:0/2635171659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64a8102eb0 0x7f64a8109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.314+0000 7f64ae5a9640 1 --2- 172.21.15.38:0/2635171659 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a81024f0 0x7f64a8102970 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.314+0000 7f64ae5a9640 1 --2- 172.21.15.38:0/2635171659 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a8101300 0x7f64a8101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.314+0000 7f64ae5a9640 1 -- 172.21.15.38:0/2635171659 >> 172.21.15.38:0/2635171659 conn(0x7f64a80fc890 msgr2=0x7f64a80fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.314+0000 7f64ae5a9640 1 -- 172.21.15.38:0/2635171659 shutdown_connections 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.314+0000 7f64ae5a9640 1 -- 172.21.15.38:0/2635171659 wait complete. 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64ae5a9640 1 Processor -- start 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64ae5a9640 1 -- start start 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64ae5a9640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a8101300 0x7f64a80750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64ae5a9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a81024f0 0x7f64a80736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64ae5a9640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64a8102eb0 0x7f64a8073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64ae5a9640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a810cf90 con 0x7f64a81024f0 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64ae5a9640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a810ce10 con 0x7f64a8102eb0 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64ae5a9640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64a810d110 con 0x7f64a8101300 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a81024f0 0x7f64a80736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a81024f0 0x7f64a80736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43348/0 (socket says 172.21.15.38:43348) 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64a77fe640 1 -- 172.21.15.38:0/1310254800 learned_addr learned my addr 172.21.15.38:0/1310254800 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64acb46640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64a8102eb0 0x7f64a8073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64a7fff640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a8101300 0x7f64a80750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64a77fe640 1 -- 172.21.15.38:0/1310254800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a8101300 msgr2=0x7f64a80750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64a77fe640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a8101300 0x7f64a80750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64a77fe640 1 -- 172.21.15.38:0/1310254800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64a8102eb0 msgr2=0x7f64a8073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64a77fe640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64a8102eb0 0x7f64a8073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64a77fe640 1 -- 172.21.15.38:0/1310254800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a80744b0 con 0x7f64a81024f0 2024-03-26T06:36:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.315+0000 7f64acb46640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64a8102eb0 0x7f64a8073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:36:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.316+0000 7f64a77fe640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a81024f0 0x7f64a80736e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f649c009230 tx=0x7f649c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.316+0000 7f64a57fa640 1 -- 172.21.15.38:0/1310254800 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c041070 con 0x7f64a81024f0 2024-03-26T06:36:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.316+0000 7f64ae5a9640 1 -- 172.21.15.38:0/1310254800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64a81a15c0 con 0x7f64a81024f0 2024-03-26T06:36:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.316+0000 7f64a57fa640 1 -- 172.21.15.38:0/1310254800 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f649c0054e0 con 0x7f64a81024f0 2024-03-26T06:36:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.316+0000 7f64a57fa640 1 -- 172.21.15.38:0/1310254800 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c0346a0 con 0x7f64a81024f0 2024-03-26T06:36:54.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.316+0000 7f64ae5a9640 1 -- 172.21.15.38:0/1310254800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64a81a1b10 con 0x7f64a81024f0 2024-03-26T06:36:54.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.318+0000 7f64a57fa640 1 -- 172.21.15.38:0/1310254800 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f649c044070 con 0x7f64a81024f0 2024-03-26T06:36:54.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.318+0000 7f64b1186740 1 Processor -- start 2024-03-26T06:36:54.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64b1186740 1 -- start start 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64a57fa640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f64900a2890 0x7f64900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64b1186740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a40e0 0x5606e37a4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64b1186740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 0x5606e37a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64b1186740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e37ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64b1186740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e37097c0 con 0x5606e37a9c40 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64b1186740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e36b2b50 con 0x5606e37a4ad0 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64b1186740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e3705860 con 0x5606e37a40e0 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64acb46640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f64900a2890 0x7f64900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64a57fa640 1 -- 172.21.15.38:0/1310254800 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f649c0f08c0 con 0x7f64a81024f0 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64acb46640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e37ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64acb46640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e37ac0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43364/0 (socket says 172.21.15.38:43364) 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64acb46640 1 -- 172.21.15.38:0/3713480543 learned_addr learned my addr 172.21.15.38:0/3713480543 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64a7fff640 1 --2- 172.21.15.38:0/3713480543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a40e0 0x5606e37a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64a77fe640 1 --2- 172.21.15.38:0/3713480543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 0x5606e37a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64acb46640 1 -- 172.21.15.38:0/3713480543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a40e0 msgr2=0x5606e37a4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64acb46640 1 --2- 172.21.15.38:0/3713480543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a40e0 0x5606e37a4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64acb46640 1 -- 172.21.15.38:0/3713480543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 msgr2=0x5606e37a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64acb46640 1 --2- 172.21.15.38:0/3713480543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 0x5606e37a95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64acb46640 1 -- 172.21.15.38:0/3713480543 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606e36c69a0 con 0x5606e37a9c40 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64a7fff640 1 --2- 172.21.15.38:0/3713480543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a40e0 0x5606e37a4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64a77fe640 1 --2- 172.21.15.38:0/3713480543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 0x5606e37a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.319+0000 7f64acb46640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f64900a2890 0x7f64900a4d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f64a0013c50 tx=0x7f64a0007e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f64acb46640 1 --2- 172.21.15.38:0/3713480543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e37ac0d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f64a0010c30 tx=0x7f64a001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa287901e5ae0b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f648e7fc640 1 -- 172.21.15.38:0/3713480543 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0021070 con 0x5606e37a9c40 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f648e7fc640 1 -- 172.21.15.38:0/3713480543 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64a0013270 con 0x5606e37a9c40 2024-03-26T06:36:54.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f648e7fc640 1 -- 172.21.15.38:0/3713480543 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0005650 con 0x5606e37a9c40 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f64b1186740 1 -- 172.21.15.38:0/3713480543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 msgr2=0x5606e37ac0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f64b1186740 1 --2- 172.21.15.38:0/3713480543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e37ac0d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f64a0010c30 tx=0x7f64a001a000 comp rx=0 tx=0).stop 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f64b1186740 1 -- 172.21.15.38:0/3713480543 shutdown_connections 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f64b1186740 1 --2- 172.21.15.38:0/3713480543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e37ac0d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f64b1186740 1 --2- 172.21.15.38:0/3713480543 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 0x5606e37a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f64b1186740 1 --2- 172.21.15.38:0/3713480543 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a40e0 0x5606e37a4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f64b1186740 1 -- 172.21.15.38:0/3713480543 >> 172.21.15.38:0/3713480543 conn(0x5606e376c400 msgr2=0x5606e379fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f64b1186740 1 -- 172.21.15.38:0/3713480543 shutdown_connections 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.320+0000 7f64b1186740 1 -- 172.21.15.38:0/3713480543 wait complete. 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64b1186740 1 -- start start 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64b1186740 1 Processor -- start 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64b1186740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a40e0 0x5606e37ae560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64b1186740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 0x5606e37aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64b1186740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a9c40 0x5606e37b5ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64b1186740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e37acb90 con 0x5606e37a40e0 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64b1186740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e36c5f90 con 0x5606e37a4ad0 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64b1186740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e37acd10 con 0x5606e37a9c40 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64b1186740 10 client.0 _ll_register_callbacks cb 0x5606e3701400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a40e0 0x5606e37ae560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a40e0 0x5606e37ae560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43380/0 (socket says 172.21.15.38:43380) 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64a7fff640 1 -- 172.21.15.38:0/4062364705 learned_addr learned my addr 172.21.15.38:0/4062364705 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64acb46640 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a9c40 0x5606e37b5ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.321+0000 7f64a77fe640 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 0x5606e37aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64a7fff640 1 -- 172.21.15.38:0/4062364705 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a9c40 msgr2=0x5606e37b5ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64a7fff640 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a9c40 0x5606e37b5ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64a7fff640 1 -- 172.21.15.38:0/4062364705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 msgr2=0x5606e37aeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64a7fff640 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 0x5606e37aeaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64a7fff640 1 -- 172.21.15.38:0/4062364705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606e341e2d0 con 0x5606e37a40e0 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64a77fe640 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 0x5606e37aeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64a7fff640 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a40e0 0x5606e37ae560 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f649800cea0 tx=0x7f649800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4701c82c702b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64877fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64877fe640 1 -- 172.21.15.38:0/4062364705 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6498011910 con 0x5606e37a40e0 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64877fe640 1 -- 172.21.15.38:0/4062364705 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f6498011ab0 con 0x5606e37a40e0 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64b1186740 1 -- 172.21.15.38:0/4062364705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f64a0013270 con 0x5606e37a40e0 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64877fe640 1 -- 172.21.15.38:0/4062364705 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6498011cb0 con 0x5606e37a40e0 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64b1186740 10 client.15054 fetch_fsmap learned FSMap version 22 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64b1186740 1 -- 172.21.15.38:0/4062364705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5606e34e86b0 con 0x5606e37a40e0 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64877fe640 1 -- 172.21.15.38:0/4062364705 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64a0013270 con 0x5606e37a40e0 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64877fe640 1 -- 172.21.15.38:0/4062364705 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6498018070 con 0x5606e37a40e0 2024-03-26T06:36:54.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64b1186740 10 client.15054 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64b1186740 10 client.15054 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64b1186740 20 client.15054 populate_metadata read hostname 'smithi038' 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64b1186740 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x5606e37bb170 0x5606e37ba1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64b1186740 4 client.15054 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.322+0000 7f64b1186740 1 -- 172.21.15.38:0/4062364705 --> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5606e3716d00 con 0x5606e37bb170 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.323+0000 7f64a77fe640 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x5606e37bb170 0x5606e37ba1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.323+0000 7f64a77fe640 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x5606e37bb170 0x5606e37ba1c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=94e07ff46a617e6e server_cookie=41946df6692df4f9 in_seq=0 out_seq=0 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.323+0000 7f64877fe640 10 client.15054 ms_handle_connect on v2:172.21.15.121:6836/1114423776 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64877fe640 1 -- 172.21.15.38:0/4062364705 <== mds.2 v2:172.21.15.121:6836/1114423776 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5606e3716d00 con 0x5606e37bb170 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64877fe640 10 client.15054 handle_command_reply: tid=0 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 1 client.15054 shutdown 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 1 -- 172.21.15.38:0/4062364705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a40e0 msgr2=0x5606e37ae560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a40e0 0x5606e37ae560 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f649800cea0 tx=0x7f649800cf90 comp rx=0 tx=0).stop 2024-03-26T06:36:54.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 1 -- 172.21.15.38:0/4062364705 shutdown_connections 2024-03-26T06:36:54.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x5606e37bb170 0x5606e37ba1c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a9c40 0x5606e37b5ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64877fe640 0 client.15054 ms_handle_reset on v2:172.21.15.121:6836/1114423776 2024-03-26T06:36:54.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a4ad0 0x5606e37aeaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 1 --2- 172.21.15.38:0/4062364705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a40e0 0x5606e37ae560 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 1 -- 172.21.15.38:0/4062364705 >> 172.21.15.38:0/4062364705 conn(0x5606e376c400 msgr2=0x5606e379ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 1 -- 172.21.15.38:0/4062364705 shutdown_connections 2024-03-26T06:36:54.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 1 -- 172.21.15.38:0/4062364705 wait complete. 2024-03-26T06:36:54.334 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.324+0000 7f64b1186740 20 client.15054 trim_cache size 0 max 16384 2024-03-26T06:36:54.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64b1186740 1 Processor -- start 2024-03-26T06:36:54.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64b1186740 1 -- start start 2024-03-26T06:36:54.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64b1186740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a40e0 0x5606e37b6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64b1186740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a4ad0 0x5606e37b5a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.344 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64b1186740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e379f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64b1186740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e37ba990 con 0x5606e37a9c40 2024-03-26T06:36:54.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64b1186740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e37baff0 con 0x5606e37a40e0 2024-03-26T06:36:54.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64b1186740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e37bb650 con 0x5606e37a4ad0 2024-03-26T06:36:54.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64acb46640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e379f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64acb46640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e379f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43394/0 (socket says 172.21.15.38:43394) 2024-03-26T06:36:54.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64acb46640 1 -- 172.21.15.38:0/822727129 learned_addr learned my addr 172.21.15.38:0/822727129 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.345 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64a7fff640 1 --2- 172.21.15.38:0/822727129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a40e0 0x5606e37b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64acb46640 1 -- 172.21.15.38:0/822727129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a4ad0 msgr2=0x5606e37b5a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64acb46640 1 --2- 172.21.15.38:0/822727129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a4ad0 0x5606e37b5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.340+0000 7f64acb46640 1 -- 172.21.15.38:0/822727129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a40e0 msgr2=0x5606e37b6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.341+0000 7f64acb46640 1 --2- 172.21.15.38:0/822727129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a40e0 0x5606e37b6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.341+0000 7f64acb46640 1 -- 172.21.15.38:0/822727129 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606e37a4eb0 con 0x5606e37a9c40 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.341+0000 7f64a7fff640 1 --2- 172.21.15.38:0/822727129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a40e0 0x5606e37b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.341+0000 7f64acb46640 1 --2- 172.21.15.38:0/822727129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e379f5b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f64a0000f80 tx=0x7f64a0010d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1fc3e4606cb9c96 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.341+0000 7f648dffb640 1 -- 172.21.15.38:0/822727129 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0021040 con 0x5606e37a9c40 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.341+0000 7f648dffb640 1 -- 172.21.15.38:0/822727129 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64a00130c0 con 0x5606e37a9c40 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.341+0000 7f648dffb640 1 -- 172.21.15.38:0/822727129 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a0005740 con 0x5606e37a9c40 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.342+0000 7f64b1186740 1 -- 172.21.15.38:0/822727129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 msgr2=0x5606e379f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.342+0000 7f64b1186740 1 --2- 172.21.15.38:0/822727129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e379f5b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f64a0000f80 tx=0x7f64a0010d00 comp rx=0 tx=0).stop 2024-03-26T06:36:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.342+0000 7f64b1186740 1 -- 172.21.15.38:0/822727129 shutdown_connections 2024-03-26T06:36:54.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.342+0000 7f64b1186740 1 --2- 172.21.15.38:0/822727129 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a9c40 0x5606e379f5b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.342+0000 7f64b1186740 1 --2- 172.21.15.38:0/822727129 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a4ad0 0x5606e37b5a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.342+0000 7f64b1186740 1 --2- 172.21.15.38:0/822727129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a40e0 0x5606e37b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.342+0000 7f64b1186740 1 -- 172.21.15.38:0/822727129 >> 172.21.15.38:0/822727129 conn(0x5606e376c400 msgr2=0x5606e37a5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.342+0000 7f64b1186740 1 -- 172.21.15.38:0/822727129 shutdown_connections 2024-03-26T06:36:54.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.342+0000 7f64b1186740 1 -- 172.21.15.38:0/822727129 wait complete. 2024-03-26T06:36:54.347 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.342+0000 7f64b1186740 1 -- start start 2024-03-26T06:36:54.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.342+0000 7f64b1186740 1 Processor -- start 2024-03-26T06:36:54.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64b1186740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a40e0 0x5606e37b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64b1186740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a4ad0 0x5606e37b3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64b1186740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a9c40 0x5606e37b4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64b1186740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e37bc310 con 0x5606e37a4ad0 2024-03-26T06:36:54.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64b1186740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e37bbcb0 con 0x5606e37a9c40 2024-03-26T06:36:54.351 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64b1186740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606e37bc970 con 0x5606e37a40e0 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64b1186740 10 client.0 _ll_register_callbacks cb 0x7f64a0013760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a4ad0 0x5606e37b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64a77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a4ad0 0x5606e37b3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43404/0 (socket says 172.21.15.38:43404) 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64a77fe640 1 -- 172.21.15.38:0/1735521455 learned_addr learned my addr 172.21.15.38:0/1735521455 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64a7fff640 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a40e0 0x5606e37b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64acb46640 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a9c40 0x5606e37b4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64a77fe640 1 -- 172.21.15.38:0/1735521455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a40e0 msgr2=0x5606e37b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64a77fe640 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a40e0 0x5606e37b2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64a77fe640 1 -- 172.21.15.38:0/1735521455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a9c40 msgr2=0x5606e37b4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64a77fe640 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a9c40 0x5606e37b4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64a77fe640 1 -- 172.21.15.38:0/1735521455 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606e37b4870 con 0x5606e37a4ad0 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64a77fe640 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a4ad0 0x5606e37b3270 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f649c0bd1a0 tx=0x7f649c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cc63eba03681167 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.352 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.343+0000 7f64877fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64877fe640 1 -- 172.21.15.38:0/1735521455 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c04b040 con 0x5606e37a4ad0 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64877fe640 1 -- 172.21.15.38:0/1735521455 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f649c0dd0b0 con 0x5606e37a4ad0 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64b1186740 1 -- 172.21.15.38:0/1735521455 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f64a00130c0 con 0x5606e37a4ad0 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64877fe640 1 -- 172.21.15.38:0/1735521455 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f649c0d60a0 con 0x5606e37a4ad0 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64b1186740 10 client.15060 fetch_fsmap learned FSMap version 22 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64b1186740 1 -- 172.21.15.38:0/1735521455 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x5606e37b5ae0 con 0x5606e37a4ad0 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64877fe640 1 -- 172.21.15.38:0/1735521455 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64a00130c0 con 0x5606e37a4ad0 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64877fe640 1 -- 172.21.15.38:0/1735521455 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f649c0b9880 con 0x5606e37a4ad0 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64b1186740 10 client.15060 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64b1186740 10 client.15060 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64b1186740 20 client.15060 populate_metadata read hostname 'smithi038' 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.345+0000 7f64b1186740 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x5606e3816dd0 0x5606e3812d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.353 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.346+0000 7f64b1186740 4 client.15060 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-26T06:36:54.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.346+0000 7f64b1186740 1 -- 172.21.15.38:0/1735521455 --> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5606e3711050 con 0x5606e3816dd0 2024-03-26T06:36:54.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.346+0000 7f64a7fff640 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x5606e3816dd0 0x5606e3812d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.346+0000 7f64a7fff640 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x5606e3816dd0 0x5606e3812d80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=92996008dc7cbf85 server_cookie=5e89447311b64b75 in_seq=0 out_seq=0 2024-03-26T06:36:54.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.346+0000 7f64877fe640 10 client.15060 ms_handle_connect on v2:172.21.15.121:6836/1114423776 2024-03-26T06:36:54.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.346+0000 7f64877fe640 1 -- 172.21.15.38:0/1735521455 <== mds.2 v2:172.21.15.121:6836/1114423776 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5606e3711050 con 0x5606e3816dd0 2024-03-26T06:36:54.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.346+0000 7f64877fe640 10 client.15060 handle_command_reply: tid=0 2024-03-26T06:36:54.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.346+0000 7f64b1186740 1 client.15060 shutdown 2024-03-26T06:36:54.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64b1186740 1 -- 172.21.15.38:0/1735521455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a4ad0 msgr2=0x5606e37b3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64b1186740 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a4ad0 0x5606e37b3270 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f649c0bd1a0 tx=0x7f649c02bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:54.354 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64b1186740 1 -- 172.21.15.38:0/1735521455 shutdown_connections 2024-03-26T06:36:54.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64b1186740 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x5606e3816dd0 0x5606e3812d80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64b1186740 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5606e37a9c40 0x5606e37b4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64b1186740 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5606e37a4ad0 0x5606e37b3270 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64877fe640 0 client.15060 ms_handle_reset on v2:172.21.15.121:6836/1114423776 2024-03-26T06:36:54.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64b1186740 1 --2- 172.21.15.38:0/1735521455 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5606e37a40e0 0x5606e37b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64b1186740 1 -- 172.21.15.38:0/1735521455 >> 172.21.15.38:0/1735521455 conn(0x5606e376c400 msgr2=0x5606e379f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64b1186740 1 -- 172.21.15.38:0/1735521455 shutdown_connections 2024-03-26T06:36:54.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64b1186740 1 -- 172.21.15.38:0/1735521455 wait complete. 2024-03-26T06:36:54.355 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64b1186740 20 client.15060 trim_cache size 0 max 16384 2024-03-26T06:36:54.356 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:36:54.356 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c2824556-eb39-11ee-95d0-87774f69a715", 2024-03-26T06:36:54.356 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 2, 2024-03-26T06:36:54.356 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24488, 2024-03-26T06:36:54.356 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-03-26T06:36:54.356 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-03-26T06:36:54.356 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-03-26T06:36:54.357 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 83.187063851000005, 2024-03-26T06:36:54.357 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 22, 2024-03-26T06:36:54.357 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 88, 2024-03-26T06:36:54.357 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 86, 2024-03-26T06:36:54.357 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 103.720940046 2024-03-26T06:36:54.357 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:36:54.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64ae5a9640 1 -- 172.21.15.38:0/1310254800 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f64900a2890 msgr2=0x7f64900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64ae5a9640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f64900a2890 0x7f64900a4d50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f64a0013c50 tx=0x7f64a0007e30 comp rx=0 tx=0).stop 2024-03-26T06:36:54.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64ae5a9640 1 -- 172.21.15.38:0/1310254800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a81024f0 msgr2=0x7f64a80736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.347+0000 7f64ae5a9640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a81024f0 0x7f64a80736e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f649c009230 tx=0x7f649c004040 comp rx=0 tx=0).stop 2024-03-26T06:36:54.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.348+0000 7f64ae5a9640 1 -- 172.21.15.38:0/1310254800 shutdown_connections 2024-03-26T06:36:54.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.348+0000 7f64ae5a9640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f64900a2890 0x7f64900a4d50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.348+0000 7f64ae5a9640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64a8102eb0 0x7f64a8073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.348+0000 7f64ae5a9640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64a81024f0 0x7f64a80736e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.348+0000 7f64ae5a9640 1 --2- 172.21.15.38:0/1310254800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f64a8101300 0x7f64a80750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.358 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.348+0000 7f64ae5a9640 1 -- 172.21.15.38:0/1310254800 >> 172.21.15.38:0/1310254800 conn(0x7f64a80fc890 msgr2=0x7f64a81057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.348+0000 7f64ae5a9640 1 -- 172.21.15.38:0/1310254800 shutdown_connections 2024-03-26T06:36:54.359 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.348+0000 7f64ae5a9640 1 -- 172.21.15.38:0/1310254800 wait complete. 2024-03-26T06:36:54.361 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-26T06:36:54.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.462+0000 7f936a80c640 1 -- 172.21.15.38:0/1911598120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93641046c0 msgr2=0x7f9364106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.462+0000 7f936a80c640 1 --2- 172.21.15.38:0/1911598120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93641046c0 0x7f9364106ab0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9358009900 tx=0x7f935802bc30 comp rx=0 tx=0).stop 2024-03-26T06:36:54.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.462+0000 7f936a80c640 1 -- 172.21.15.38:0/1911598120 shutdown_connections 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.462+0000 7f936a80c640 1 --2- 172.21.15.38:0/1911598120 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9364106ff0 0x7f9364109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.462+0000 7f936a80c640 1 --2- 172.21.15.38:0/1911598120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f93641046c0 0x7f9364106ab0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.462+0000 7f936a80c640 1 --2- 172.21.15.38:0/1911598120 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9364101d90 0x7f9364104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.462+0000 7f936a80c640 1 -- 172.21.15.38:0/1911598120 >> 172.21.15.38:0/1911598120 conn(0x7f93640fb7a0 msgr2=0x7f93640fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.462+0000 7f936a80c640 1 -- 172.21.15.38:0/1911598120 shutdown_connections 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.462+0000 7f936a80c640 1 -- 172.21.15.38:0/1911598120 wait complete. 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.462+0000 7f936a80c640 1 Processor -- start 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.462+0000 7f936a80c640 1 -- start start 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f936a80c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9364101d90 0x7f9364197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f936a80c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93641046c0 0x7f9364197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f936a80c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9364106ff0 0x7f936419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f936a80c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936410cfa0 con 0x7f9364101d90 2024-03-26T06:36:54.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f936a80c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936410ce20 con 0x7f93641046c0 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f936a80c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936410d120 con 0x7f9364106ff0 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f9363fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9364101d90 0x7f9364197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f9363fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9364101d90 0x7f9364197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43434/0 (socket says 172.21.15.38:43434) 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f9363fff640 1 -- 172.21.15.38:0/1796143806 learned_addr learned my addr 172.21.15.38:0/1796143806 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f9368da9640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9364106ff0 0x7f936419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f93637fe640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93641046c0 0x7f9364197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f9363fff640 1 -- 172.21.15.38:0/1796143806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9364106ff0 msgr2=0x7f936419ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f9363fff640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9364106ff0 0x7f936419ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f9363fff640 1 -- 172.21.15.38:0/1796143806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93641046c0 msgr2=0x7f9364197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f9363fff640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93641046c0 0x7f9364197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f9363fff640 1 -- 172.21.15.38:0/1796143806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936419f280 con 0x7f9364101d90 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f9363fff640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9364101d90 0x7f9364197660 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f935400d950 tx=0x7f935400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f93617fa640 1 -- 172.21.15.38:0/1796143806 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354017070 con 0x7f9364101d90 2024-03-26T06:36:54.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f936a80c640 1 -- 172.21.15.38:0/1796143806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936419f570 con 0x7f9364101d90 2024-03-26T06:36:54.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f93617fa640 1 -- 172.21.15.38:0/1796143806 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9354009c30 con 0x7f9364101d90 2024-03-26T06:36:54.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f93617fa640 1 -- 172.21.15.38:0/1796143806 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935400f6b0 con 0x7f9364101d90 2024-03-26T06:36:54.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.463+0000 7f936a80c640 1 -- 172.21.15.38:0/1796143806 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936419fa90 con 0x7f9364101d90 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.465+0000 7f93617fa640 1 -- 172.21.15.38:0/1796143806 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f93540109d0 con 0x7f9364101d90 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f936d3df740 1 Processor -- start 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f936d3df740 1 -- start start 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f936d3df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10d4f0 0x55c18d10d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f936d3df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d1129e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f936d3df740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d113050 0x55c18d1154e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f936d3df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d06e7c0 con 0x55c18d10dee0 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f936d3df740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d017b50 con 0x55c18d113050 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f936d3df740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d06a860 con 0x55c18d10d4f0 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f9363fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d1129e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f93617fa640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f934c0a2890 0x7f934c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f9363fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d1129e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43444/0 (socket says 172.21.15.38:43444) 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f9363fff640 1 -- 172.21.15.38:0/1681776161 learned_addr learned my addr 172.21.15.38:0/1681776161 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f93637fe640 1 --2- 172.21.15.38:0/1681776161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d113050 0x55c18d1154e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f9368da9640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f934c0a2890 0x7f934c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f9368da9640 1 --2- 172.21.15.38:0/1681776161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10d4f0 0x55c18d10d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f93617fa640 1 -- 172.21.15.38:0/1796143806 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f93540ce7f0 con 0x7f9364101d90 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f9363fff640 1 -- 172.21.15.38:0/1681776161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10d4f0 msgr2=0x55c18d10d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f9363fff640 1 --2- 172.21.15.38:0/1681776161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10d4f0 0x55c18d10d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f9363fff640 1 -- 172.21.15.38:0/1681776161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d113050 msgr2=0x55c18d1154e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f9363fff640 1 --2- 172.21.15.38:0/1681776161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d113050 0x55c18d1154e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f9363fff640 1 -- 172.21.15.38:0/1681776161 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c18d02b9a0 con 0x55c18d10dee0 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.466+0000 7f93637fe640 1 --2- 172.21.15.38:0/1681776161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d113050 0x55c18d1154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f9363fff640 1 --2- 172.21.15.38:0/1681776161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d1129e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9354018000 tx=0x7f9354005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6adc1e1b070a3aa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f9368da9640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f934c0a2890 0x7f934c0a4d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x55c18d10e740 tx=0x7f935c009b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f934a7fc640 1 -- 172.21.15.38:0/1681776161 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354020070 con 0x55c18d10dee0 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f934a7fc640 1 -- 172.21.15.38:0/1681776161 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f935409f140 con 0x55c18d10dee0 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f934a7fc640 1 -- 172.21.15.38:0/1681776161 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93540cee40 con 0x55c18d10dee0 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f936d3df740 1 -- 172.21.15.38:0/1681776161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 msgr2=0x55c18d1129e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f936d3df740 1 --2- 172.21.15.38:0/1681776161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d1129e0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9354018000 tx=0x7f9354005880 comp rx=0 tx=0).stop 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f936d3df740 1 -- 172.21.15.38:0/1681776161 shutdown_connections 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f936d3df740 1 --2- 172.21.15.38:0/1681776161 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d113050 0x55c18d1154e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f936d3df740 1 --2- 172.21.15.38:0/1681776161 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d1129e0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f936d3df740 1 --2- 172.21.15.38:0/1681776161 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10d4f0 0x55c18d10d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f936d3df740 1 -- 172.21.15.38:0/1681776161 >> 172.21.15.38:0/1681776161 conn(0x55c18d0d1400 msgr2=0x55c18d104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f936d3df740 1 -- 172.21.15.38:0/1681776161 shutdown_connections 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.467+0000 7f936d3df740 1 -- 172.21.15.38:0/1681776161 wait complete. 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f936d3df740 1 -- start start 2024-03-26T06:36:54.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f936d3df740 1 Processor -- start 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f936d3df740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d10d4f0 0x55c18d117a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f936d3df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d117f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f936d3df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d113050 0x55c18d11ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f936d3df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d115fa0 con 0x55c18d10dee0 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f936d3df740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d02af90 con 0x55c18d10d4f0 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f936d3df740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d116120 con 0x55c18d113050 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f93637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d117f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f93637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d117f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43458/0 (socket says 172.21.15.38:43458) 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f93637fe640 1 -- 172.21.15.38:0/3561067438 learned_addr learned my addr 172.21.15.38:0/3561067438 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f936d3df740 10 client.0 _ll_register_callbacks cb 0x55c18d066400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f9363fff640 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d10d4f0 0x55c18d117a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f9368da9640 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d113050 0x55c18d11ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f93637fe640 1 -- 172.21.15.38:0/3561067438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d113050 msgr2=0x55c18d11ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f93637fe640 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d113050 0x55c18d11ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f93637fe640 1 -- 172.21.15.38:0/3561067438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d10d4f0 msgr2=0x55c18d117a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.468+0000 7f93637fe640 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d10d4f0 0x55c18d117a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f93637fe640 1 -- 172.21.15.38:0/3561067438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c18cd832d0 con 0x55c18d10dee0 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f9368da9640 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d113050 0x55c18d11ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f93637fe640 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d117f40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f935802c7b0 tx=0x7f9358004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428f510b13e7590d server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f93437fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f93437fe640 1 -- 172.21.15.38:0/3561067438 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9358005230 con 0x55c18d10dee0 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f93437fe640 1 -- 172.21.15.38:0/3561067438 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f93580053d0 con 0x55c18d10dee0 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f936d3df740 1 -- 172.21.15.38:0/3561067438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f935409f140 con 0x55c18d10dee0 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f93437fe640 1 -- 172.21.15.38:0/3561067438 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93580055d0 con 0x55c18d10dee0 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f936d3df740 10 client.15072 fetch_fsmap learned FSMap version 22 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f936d3df740 1 -- 172.21.15.38:0/3561067438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55c18ce4d6b0 con 0x55c18d10dee0 2024-03-26T06:36:54.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f93437fe640 1 -- 172.21.15.38:0/3561067438 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f935409f140 con 0x55c18d10dee0 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f93437fe640 1 -- 172.21.15.38:0/3561067438 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9358003bc0 con 0x55c18d10dee0 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f936d3df740 10 client.15072 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f936d3df740 10 client.15072 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f936d3df740 20 client.15072 populate_metadata read hostname 'smithi038' 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f936d3df740 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x55c18d1245b0 0x55c18d123680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f936d3df740 4 client.15072 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.469+0000 7f936d3df740 1 -- 172.21.15.38:0/3561067438 --> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c18d07bd00 con 0x55c18d1245b0 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.470+0000 7f9363fff640 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x55c18d1245b0 0x55c18d123680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.470+0000 7f9363fff640 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x55c18d1245b0 0x55c18d123680 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=20a6469706305170 server_cookie=4ae9ebbb5b406a24 in_seq=0 out_seq=0 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.470+0000 7f93437fe640 10 client.15072 ms_handle_connect on v2:172.21.15.170:6838/2204378068 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f93437fe640 1 -- 172.21.15.38:0/3561067438 <== mds.1 v2:172.21.15.170:6838/2204378068 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c18d07bd00 con 0x55c18d1245b0 2024-03-26T06:36:54.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f93437fe640 10 client.15072 handle_command_reply: tid=0 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 1 client.15072 shutdown 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 1 -- 172.21.15.38:0/3561067438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 msgr2=0x55c18d117f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d117f40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f935802c7b0 tx=0x7f9358004270 comp rx=0 tx=0).stop 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 1 -- 172.21.15.38:0/3561067438 shutdown_connections 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x55c18d1245b0 0x55c18d123680 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d113050 0x55c18d11ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d117f40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f93437fe640 0 client.15072 ms_handle_reset on v2:172.21.15.170:6838/2204378068 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 1 --2- 172.21.15.38:0/3561067438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d10d4f0 0x55c18d117a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 1 -- 172.21.15.38:0/3561067438 >> 172.21.15.38:0/3561067438 conn(0x55c18d0d1400 msgr2=0x55c18d103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 1 -- 172.21.15.38:0/3561067438 shutdown_connections 2024-03-26T06:36:54.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 1 -- 172.21.15.38:0/3561067438 wait complete. 2024-03-26T06:36:54.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.471+0000 7f936d3df740 20 client.15072 trim_cache size 0 max 16384 2024-03-26T06:36:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f936d3df740 1 Processor -- start 2024-03-26T06:36:54.494 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f936d3df740 1 -- start start 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f936d3df740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d0d1400 0x55c18d10f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f936d3df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10d4f0 0x55c18d11ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f936d3df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d10f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f936d3df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d123dd0 con 0x55c18d10dee0 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f936d3df740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d124430 con 0x55c18d0d1400 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f936d3df740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d124a90 con 0x55c18d10d4f0 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f9368da9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d10f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f9368da9640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d10f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58068/0 (socket says 172.21.15.38:58068) 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f9368da9640 1 -- 172.21.15.38:0/3251615970 learned_addr learned my addr 172.21.15.38:0/3251615970 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f9363fff640 1 --2- 172.21.15.38:0/3251615970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d0d1400 0x55c18d10f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f9368da9640 1 -- 172.21.15.38:0/3251615970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10d4f0 msgr2=0x55c18d11ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f9368da9640 1 --2- 172.21.15.38:0/3251615970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10d4f0 0x55c18d11ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f9368da9640 1 -- 172.21.15.38:0/3251615970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d0d1400 msgr2=0x55c18d10f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f9368da9640 1 --2- 172.21.15.38:0/3251615970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d0d1400 0x55c18d10f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f9368da9640 1 -- 172.21.15.38:0/3251615970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c18d0ffb20 con 0x55c18d10dee0 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.487+0000 7f9368da9640 1 --2- 172.21.15.38:0/3251615970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d10f980 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f935c0121e0 tx=0x7f935c009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e9c953439d0470 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f9349ffb640 1 -- 172.21.15.38:0/3251615970 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935c012210 con 0x55c18d10dee0 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f9349ffb640 1 -- 172.21.15.38:0/3251615970 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f935c004030 con 0x55c18d10dee0 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f9349ffb640 1 -- 172.21.15.38:0/3251615970 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f935c005280 con 0x55c18d10dee0 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f936d3df740 1 -- 172.21.15.38:0/3251615970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 msgr2=0x55c18d10f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f936d3df740 1 --2- 172.21.15.38:0/3251615970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d10f980 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f935c0121e0 tx=0x7f935c009600 comp rx=0 tx=0).stop 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f936d3df740 1 -- 172.21.15.38:0/3251615970 shutdown_connections 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f936d3df740 1 --2- 172.21.15.38:0/3251615970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d10dee0 0x55c18d10f980 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f936d3df740 1 --2- 172.21.15.38:0/3251615970 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10d4f0 0x55c18d11ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.496 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f936d3df740 1 --2- 172.21.15.38:0/3251615970 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d0d1400 0x55c18d10f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f936d3df740 1 -- 172.21.15.38:0/3251615970 >> 172.21.15.38:0/3251615970 conn(0x55c18d1239f0 msgr2=0x55c18d10e980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f936d3df740 1 -- 172.21.15.38:0/3251615970 shutdown_connections 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f936d3df740 1 -- 172.21.15.38:0/3251615970 wait complete. 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.488+0000 7f936d3df740 1 -- start start 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f936d3df740 1 Processor -- start 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f936d3df740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d0d1400 0x55c18d11bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f936d3df740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d10d4f0 0x55c18d11c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f936d3df740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10dee0 0x55c18d17a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f936d3df740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d125750 con 0x55c18d0d1400 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f936d3df740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d1250f0 con 0x55c18d10d4f0 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f936d3df740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c18d125db0 con 0x55c18d10dee0 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f936d3df740 10 client.0 _ll_register_callbacks cb 0x7f935409aef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:36:54.497 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f9363fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d0d1400 0x55c18d11bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f9363fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d0d1400 0x55c18d11bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58080/0 (socket says 172.21.15.38:58080) 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f9363fff640 1 -- 172.21.15.38:0/3005509731 learned_addr learned my addr 172.21.15.38:0/3005509731 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f93637fe640 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d10d4f0 0x55c18d11c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f9363fff640 1 -- 172.21.15.38:0/3005509731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10dee0 msgr2=0x55c18d17a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f9363fff640 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10dee0 0x55c18d17a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f9363fff640 1 -- 172.21.15.38:0/3005509731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d10d4f0 msgr2=0x55c18d11c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f9363fff640 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d10d4f0 0x55c18d11c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f9363fff640 1 -- 172.21.15.38:0/3005509731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c18d11f1f0 con 0x55c18d0d1400 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f9363fff640 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d0d1400 0x55c18d11bc50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9354097a70 tx=0x7f93540cc3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bab5c5d0ff55da server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f93437fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.489+0000 7f93437fe640 1 -- 172.21.15.38:0/3005509731 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9354020040 con 0x55c18d0d1400 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f93437fe640 1 -- 172.21.15.38:0/3005509731 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f93540cebe0 con 0x55c18d0d1400 2024-03-26T06:36:54.498 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f936d3df740 1 -- 172.21.15.38:0/3005509731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f935c004030 con 0x55c18d0d1400 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f93437fe640 1 -- 172.21.15.38:0/3005509731 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93540cc420 con 0x55c18d0d1400 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f936d3df740 10 client.15078 fetch_fsmap learned FSMap version 22 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f936d3df740 1 -- 172.21.15.38:0/3005509731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55c18d17b720 con 0x55c18d0d1400 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f93437fe640 1 -- 172.21.15.38:0/3005509731 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f935c004030 con 0x55c18d0d1400 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f93437fe640 1 -- 172.21.15.38:0/3005509731 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f935409a4f0 con 0x55c18d0d1400 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f936d3df740 10 client.15078 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f936d3df740 10 client.15078 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f936d3df740 20 client.15078 populate_metadata read hostname 'smithi038' 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f936d3df740 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x55c18d113050 0x55c18d17b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f936d3df740 4 client.15078 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f936d3df740 1 -- 172.21.15.38:0/3005509731 --> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c18d076050 con 0x55c18d113050 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.490+0000 7f93637fe640 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x55c18d113050 0x55c18d17b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f93637fe640 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x55c18d113050 0x55c18d17b8f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1aaa70f1ed10d8f5 server_cookie=55bbcad7e1199cd in_seq=0 out_seq=0 2024-03-26T06:36:54.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f93437fe640 10 client.15078 ms_handle_connect on v2:172.21.15.170:6838/2204378068 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f93437fe640 1 -- 172.21.15.38:0/3005509731 <== mds.1 v2:172.21.15.170:6838/2204378068 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55c18d076050 con 0x55c18d113050 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f93437fe640 10 client.15078 handle_command_reply: tid=0 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 1 client.15078 shutdown 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 1 -- 172.21.15.38:0/3005509731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d0d1400 msgr2=0x55c18d11bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d0d1400 0x55c18d11bc50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9354097a70 tx=0x7f93540cc3b0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 1 -- 172.21.15.38:0/3005509731 shutdown_connections 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x55c18d113050 0x55c18d17b8f0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55c18d10dee0 0x55c18d17a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55c18d10d4f0 0x55c18d11c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 1 --2- 172.21.15.38:0/3005509731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c18d0d1400 0x55c18d11bc50 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f93437fe640 0 client.15078 ms_handle_reset on v2:172.21.15.170:6838/2204378068 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 1 -- 172.21.15.38:0/3005509731 >> 172.21.15.38:0/3005509731 conn(0x55c18d1239f0 msgr2=0x55c18d10f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 1 -- 172.21.15.38:0/3005509731 shutdown_connections 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 1 -- 172.21.15.38:0/3005509731 wait complete. 2024-03-26T06:36:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.491+0000 7f936d3df740 20 client.15078 trim_cache size 0 max 16384 2024-03-26T06:36:54.501 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:36:54.501 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c2824556-eb39-11ee-95d0-87774f69a715", 2024-03-26T06:36:54.501 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 1, 2024-03-26T06:36:54.501 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24502, 2024-03-26T06:36:54.501 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-03-26T06:36:54.501 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-03-26T06:36:54.501 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-03-26T06:36:54.501 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 84.343167868999998, 2024-03-26T06:36:54.501 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 22, 2024-03-26T06:36:54.501 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 88, 2024-03-26T06:36:54.502 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 86, 2024-03-26T06:36:54.502 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 101.160709811 2024-03-26T06:36:54.502 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:36:54.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 -- 172.21.15.38:0/1796143806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f934c0a2890 msgr2=0x7f934c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f934c0a2890 0x7f934c0a4d50 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x55c18d10e740 tx=0x7f935c009b50 comp rx=0 tx=0).stop 2024-03-26T06:36:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 -- 172.21.15.38:0/1796143806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9364101d90 msgr2=0x7f9364197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9364101d90 0x7f9364197660 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f935400d950 tx=0x7f935400de10 comp rx=0 tx=0).stop 2024-03-26T06:36:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 -- 172.21.15.38:0/1796143806 shutdown_connections 2024-03-26T06:36:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f934c0a2890 0x7f934c0a4d50 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9364106ff0 0x7f936419ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93641046c0 0x7f9364197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 --2- 172.21.15.38:0/1796143806 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9364101d90 0x7f9364197660 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 -- 172.21.15.38:0/1796143806 >> 172.21.15.38:0/1796143806 conn(0x7f93640fb7a0 msgr2=0x7f9364102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 -- 172.21.15.38:0/1796143806 shutdown_connections 2024-03-26T06:36:54.503 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.492+0000 7f936a80c640 1 -- 172.21.15.38:0/1796143806 wait complete. 2024-03-26T06:36:54.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:54 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1459136603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:36:54.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:54 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/258703971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:36:54.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:54 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3183548765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:36:54.505 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-26T06:36:54.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.606+0000 7f26ae4f8640 1 -- 172.21.15.38:0/2238384232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a8102eb0 msgr2=0x7f26a8109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.606+0000 7f26ae4f8640 1 --2- 172.21.15.38:0/2238384232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a8102eb0 0x7f26a8109740 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009ca0 tx=0x7f26a002c0f0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.606+0000 7f26ae4f8640 1 -- 172.21.15.38:0/2238384232 shutdown_connections 2024-03-26T06:36:54.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.606+0000 7f26ae4f8640 1 --2- 172.21.15.38:0/2238384232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a8102eb0 0x7f26a8109740 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.606+0000 7f26ae4f8640 1 --2- 172.21.15.38:0/2238384232 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26a81024f0 0x7f26a8102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.606+0000 7f26ae4f8640 1 --2- 172.21.15.38:0/2238384232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26a8101300 0x7f26a8101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.606+0000 7f26ae4f8640 1 -- 172.21.15.38:0/2238384232 >> 172.21.15.38:0/2238384232 conn(0x7f26a80fc890 msgr2=0x7f26a80fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.606+0000 7f26ae4f8640 1 -- 172.21.15.38:0/2238384232 shutdown_connections 2024-03-26T06:36:54.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.606+0000 7f26ae4f8640 1 -- 172.21.15.38:0/2238384232 wait complete. 2024-03-26T06:36:54.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.606+0000 7f26ae4f8640 1 Processor -- start 2024-03-26T06:36:54.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.606+0000 7f26ae4f8640 1 -- start start 2024-03-26T06:36:54.611 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26ae4f8640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a8101300 0x7f26a8197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26ae4f8640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26a81024f0 0x7f26a8197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26ae4f8640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26a8102eb0 0x7f26a819eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26ae4f8640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a810cf90 con 0x7f26a8101300 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26ae4f8640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a810ce10 con 0x7f26a8102eb0 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26ae4f8640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a810d110 con 0x7f26a81024f0 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a8101300 0x7f26a8197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a8101300 0x7f26a8197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58106/0 (socket says 172.21.15.38:58106) 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a7fff640 1 -- 172.21.15.38:0/3231927381 learned_addr learned my addr 172.21.15.38:0/3231927381 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26aca95640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26a8102eb0 0x7f26a819eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a77fe640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26a81024f0 0x7f26a8197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a7fff640 1 -- 172.21.15.38:0/3231927381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26a81024f0 msgr2=0x7f26a8197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a7fff640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26a81024f0 0x7f26a8197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a7fff640 1 -- 172.21.15.38:0/3231927381 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26a8102eb0 msgr2=0x7f26a819eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a7fff640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26a8102eb0 0x7f26a819eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a7fff640 1 -- 172.21.15.38:0/3231927381 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a819f150 con 0x7f26a8101300 2024-03-26T06:36:54.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a7fff640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a8101300 0x7f26a8197530 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f269800e9c0 tx=0x7f269800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a57fa640 1 -- 172.21.15.38:0/3231927381 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269800ccc0 con 0x7f26a8101300 2024-03-26T06:36:54.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26a57fa640 1 -- 172.21.15.38:0/3231927381 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26980035f0 con 0x7f26a8101300 2024-03-26T06:36:54.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.607+0000 7f26ae4f8640 1 -- 172.21.15.38:0/3231927381 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a819f440 con 0x7f26a8101300 2024-03-26T06:36:54.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.608+0000 7f26a57fa640 1 -- 172.21.15.38:0/3231927381 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269800ce60 con 0x7f26a8101300 2024-03-26T06:36:54.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.608+0000 7f26ae4f8640 1 -- 172.21.15.38:0/3231927381 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a819f990 con 0x7f26a8101300 2024-03-26T06:36:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26a57fa640 1 -- 172.21.15.38:0/3231927381 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f26980037a0 con 0x7f26a8101300 2024-03-26T06:36:54.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26b11bb740 1 Processor -- start 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26b11bb740 1 -- start start 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26a57fa640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f26900a2890 0x7f26900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e4f7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e4fc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4fcc40 0x556b7e4ff0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26b11bb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e45c7c0 con 0x556b7e4f70e0 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26b11bb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e405b50 con 0x556b7e4f7ad0 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26b11bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e458860 con 0x556b7e4fcc40 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26aca95640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f26900a2890 0x7f26900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.610+0000 7f26a57fa640 1 -- 172.21.15.38:0/3231927381 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f26980cef50 con 0x7f26a8101300 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e4f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e4f7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58114/0 (socket says 172.21.15.38:58114) 2024-03-26T06:36:54.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26a7fff640 1 -- 172.21.15.38:0/1071881773 learned_addr learned my addr 172.21.15.38:0/1071881773 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26a77fe640 1 --2- 172.21.15.38:0/1071881773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e4fc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26aca95640 1 --2- 172.21.15.38:0/1071881773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4fcc40 0x556b7e4ff0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26a7fff640 1 -- 172.21.15.38:0/1071881773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4fcc40 msgr2=0x556b7e4ff0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26a7fff640 1 --2- 172.21.15.38:0/1071881773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4fcc40 0x556b7e4ff0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26a7fff640 1 -- 172.21.15.38:0/1071881773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 msgr2=0x556b7e4fc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26a7fff640 1 --2- 172.21.15.38:0/1071881773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e4fc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26a7fff640 1 -- 172.21.15.38:0/1071881773 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b7e4199a0 con 0x556b7e4f70e0 2024-03-26T06:36:54.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26aca95640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f26900a2890 0x7f26900a4d50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f26a0005e30 tx=0x7f26a002cca0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f26a7fff640 1 --2- 172.21.15.38:0/1071881773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e4f7500 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f269800c6a0 tx=0x7f269809b0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c43c43cb101fd375 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f268e7fc640 1 -- 172.21.15.38:0/1071881773 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26980b3b80 con 0x556b7e4f70e0 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f268e7fc640 1 -- 172.21.15.38:0/1071881773 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f269809be90 con 0x556b7e4f70e0 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.611+0000 7f268e7fc640 1 -- 172.21.15.38:0/1071881773 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2698003b30 con 0x556b7e4f70e0 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 -- 172.21.15.38:0/1071881773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 msgr2=0x556b7e4f7500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 --2- 172.21.15.38:0/1071881773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e4f7500 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f269800c6a0 tx=0x7f269809b0f0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 -- 172.21.15.38:0/1071881773 shutdown_connections 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 --2- 172.21.15.38:0/1071881773 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4fcc40 0x556b7e4ff0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 --2- 172.21.15.38:0/1071881773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e4fc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 --2- 172.21.15.38:0/1071881773 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e4f7500 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 -- 172.21.15.38:0/1071881773 >> 172.21.15.38:0/1071881773 conn(0x556b7e4bf400 msgr2=0x556b7e4f2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 -- 172.21.15.38:0/1071881773 shutdown_connections 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 -- 172.21.15.38:0/1071881773 wait complete. 2024-03-26T06:36:54.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 -- start start 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 Processor -- start 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.612+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4f70e0 0x556b7e5014e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e501a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26b11bb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e4ffbf0 con 0x556b7e4fcc40 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26b11bb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e418f90 con 0x556b7e4f7ad0 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26b11bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e4ffd70 con 0x556b7e4f70e0 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26b11bb740 10 client.0 _ll_register_callbacks cb 0x556b7e454400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26aca95640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26aca95640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58116/0 (socket says 172.21.15.38:58116) 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26aca95640 1 -- 172.21.15.38:0/1952098145 learned_addr learned my addr 172.21.15.38:0/1952098145 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26a7fff640 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4f70e0 0x556b7e5014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26a77fe640 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e501a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26aca95640 1 -- 172.21.15.38:0/1952098145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4f70e0 msgr2=0x556b7e5014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26aca95640 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4f70e0 0x556b7e5014e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26aca95640 1 -- 172.21.15.38:0/1952098145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 msgr2=0x556b7e501a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26aca95640 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e501a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26aca95640 1 -- 172.21.15.38:0/1952098145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b7e1712d0 con 0x556b7e4fcc40 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26a77fe640 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e501a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26aca95640 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f26a0006010 tx=0x7f26a002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bd28bd2c8a47bb server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26877fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26877fe640 1 -- 172.21.15.38:0/1952098145 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0039040 con 0x556b7e4fcc40 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26877fe640 1 -- 172.21.15.38:0/1952098145 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f26a0003e00 con 0x556b7e4fcc40 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26b11bb740 1 -- 172.21.15.38:0/1952098145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f269809be90 con 0x556b7e4fcc40 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.613+0000 7f26877fe640 1 -- 172.21.15.38:0/1952098145 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26a0004030 con 0x556b7e4fcc40 2024-03-26T06:36:54.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.614+0000 7f26b11bb740 10 client.15090 fetch_fsmap learned FSMap version 22 2024-03-26T06:36:54.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.614+0000 7f26b11bb740 1 -- 172.21.15.38:0/1952098145 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x556b7e23b6b0 con 0x556b7e4fcc40 2024-03-26T06:36:54.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.614+0000 7f26877fe640 1 -- 172.21.15.38:0/1952098145 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f269809be90 con 0x556b7e4fcc40 2024-03-26T06:36:54.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.614+0000 7f26877fe640 1 -- 172.21.15.38:0/1952098145 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f26a003f070 con 0x556b7e4fcc40 2024-03-26T06:36:54.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.614+0000 7f26b11bb740 10 client.15090 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:36:54.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.614+0000 7f26b11bb740 10 client.15090 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-26T06:36:54.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.614+0000 7f26b11bb740 20 client.15090 populate_metadata read hostname 'smithi038' 2024-03-26T06:36:54.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.614+0000 7f26b11bb740 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556b7e50e0d0 0x556b7e50d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.614+0000 7f26b11bb740 4 client.15090 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:36:54.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.614+0000 7f26b11bb740 1 -- 172.21.15.38:0/1952098145 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b7e469d00 con 0x556b7e50e0d0 2024-03-26T06:36:54.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.614+0000 7f26a7fff640 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556b7e50e0d0 0x556b7e50d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26877fe640 10 client.15090 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:36:54.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26a7fff640 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556b7e50e0d0 0x556b7e50d130 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2abcb8d6ed4b1ce server_cookie=eed8dcd283b3537b in_seq=0 out_seq=0 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26877fe640 1 -- 172.21.15.38:0/1952098145 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556b7e469d00 con 0x556b7e50e0d0 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26877fe640 10 client.15090 handle_command_reply: tid=0 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26b11bb740 1 client.15090 shutdown 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26b11bb740 1 -- 172.21.15.38:0/1952098145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 msgr2=0x556b7e508a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26b11bb740 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a60 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f26a0006010 tx=0x7f26a002c0f0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26b11bb740 1 -- 172.21.15.38:0/1952098145 shutdown_connections 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26b11bb740 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556b7e50e0d0 0x556b7e50d130 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26b11bb740 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a60 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26b11bb740 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e501a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26877fe640 0 client.15090 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26b11bb740 1 --2- 172.21.15.38:0/1952098145 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4f70e0 0x556b7e5014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.615+0000 7f26b11bb740 1 -- 172.21.15.38:0/1952098145 >> 172.21.15.38:0/1952098145 conn(0x556b7e4bf400 msgr2=0x556b7e4f1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.616+0000 7f26b11bb740 1 -- 172.21.15.38:0/1952098145 shutdown_connections 2024-03-26T06:36:54.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.616+0000 7f26b11bb740 1 -- 172.21.15.38:0/1952098145 wait complete. 2024-03-26T06:36:54.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.616+0000 7f26b11bb740 20 client.15090 trim_cache size 0 max 16384 2024-03-26T06:36:54.639 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.631+0000 7f26b11bb740 1 Processor -- start 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.631+0000 7f26b11bb740 1 -- start start 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.631+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f70e0 0x556b7e4f23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.631+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4f7ad0 0x556b7e4f2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.631+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.631+0000 7f26b11bb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e50d8f0 con 0x556b7e4fcc40 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.631+0000 7f26b11bb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e50df50 con 0x556b7e4f70e0 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.631+0000 7f26b11bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e50e5b0 con 0x556b7e4f7ad0 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26aca95640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26aca95640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58126/0 (socket says 172.21.15.38:58126) 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26aca95640 1 -- 172.21.15.38:0/2728954424 learned_addr learned my addr 172.21.15.38:0/2728954424 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26a77fe640 1 --2- 172.21.15.38:0/2728954424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4f7ad0 0x556b7e4f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.640 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26a7fff640 1 --2- 172.21.15.38:0/2728954424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f70e0 0x556b7e4f23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26aca95640 1 -- 172.21.15.38:0/2728954424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4f7ad0 msgr2=0x556b7e4f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26aca95640 1 --2- 172.21.15.38:0/2728954424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4f7ad0 0x556b7e4f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26aca95640 1 -- 172.21.15.38:0/2728954424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f70e0 msgr2=0x556b7e4f23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26aca95640 1 --2- 172.21.15.38:0/2728954424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f70e0 0x556b7e4f23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26aca95640 1 -- 172.21.15.38:0/2728954424 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b7e4f37a0 con 0x556b7e4fcc40 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26a7fff640 1 --2- 172.21.15.38:0/2728954424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f70e0 0x556b7e4f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f26aca95640 1 --2- 172.21.15.38:0/2728954424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f26a0032bc0 tx=0x7f26a0005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dffacbe594d918b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f268dffb640 1 -- 172.21.15.38:0/2728954424 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0039070 con 0x556b7e4fcc40 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f268dffb640 1 -- 172.21.15.38:0/2728954424 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26a0052070 con 0x556b7e4fcc40 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.632+0000 7f268dffb640 1 -- 172.21.15.38:0/2728954424 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a0004810 con 0x556b7e4fcc40 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.633+0000 7f26b11bb740 1 -- 172.21.15.38:0/2728954424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 msgr2=0x556b7e508a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.633+0000 7f26b11bb740 1 --2- 172.21.15.38:0/2728954424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f26a0032bc0 tx=0x7f26a0005d80 comp rx=0 tx=0).stop 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.633+0000 7f26b11bb740 1 -- 172.21.15.38:0/2728954424 shutdown_connections 2024-03-26T06:36:54.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.633+0000 7f26b11bb740 1 --2- 172.21.15.38:0/2728954424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4fcc40 0x556b7e508a20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.633+0000 7f26b11bb740 1 --2- 172.21.15.38:0/2728954424 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4f7ad0 0x556b7e4f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.633+0000 7f26b11bb740 1 --2- 172.21.15.38:0/2728954424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f70e0 0x556b7e4f23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.633+0000 7f26b11bb740 1 -- 172.21.15.38:0/2728954424 >> 172.21.15.38:0/2728954424 conn(0x556b7e4bf400 msgr2=0x556b7e50d130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.633+0000 7f26b11bb740 1 -- 172.21.15.38:0/2728954424 shutdown_connections 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.633+0000 7f26b11bb740 1 -- 172.21.15.38:0/2728954424 wait complete. 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.633+0000 7f26b11bb740 1 -- start start 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.633+0000 7f26b11bb740 1 Processor -- start 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e505760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e505ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26b11bb740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4fcc40 0x556b7e563cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26b11bb740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e50f270 con 0x556b7e4f70e0 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26b11bb740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e50ec10 con 0x556b7e4f7ad0 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26b11bb740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b7e50f8d0 con 0x556b7e4fcc40 2024-03-26T06:36:54.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26b11bb740 10 client.0 _ll_register_callbacks cb 0x7f2698005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e505760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26a7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e505760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58140/0 (socket says 172.21.15.38:58140) 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26a7fff640 1 -- 172.21.15.38:0/3152509194 learned_addr learned my addr 172.21.15.38:0/3152509194 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26a77fe640 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e505ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26aca95640 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4fcc40 0x556b7e563cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26a7fff640 1 -- 172.21.15.38:0/3152509194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4fcc40 msgr2=0x556b7e563cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26a7fff640 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4fcc40 0x556b7e563cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26a7fff640 1 -- 172.21.15.38:0/3152509194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 msgr2=0x556b7e505ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26a7fff640 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e505ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26a7fff640 1 -- 172.21.15.38:0/3152509194 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b7e5062e0 con 0x556b7e4f70e0 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26aca95640 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4fcc40 0x556b7e563cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26a7fff640 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e505760 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f26980058c0 tx=0x7f2698097b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577b0b33404df7b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:54.643 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26877fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26877fe640 1 -- 172.21.15.38:0/3152509194 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269809be90 con 0x556b7e4f70e0 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26877fe640 1 -- 172.21.15.38:0/3152509194 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f2698099a90 con 0x556b7e4f70e0 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26b11bb740 1 -- 172.21.15.38:0/3152509194 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f26a0052070 con 0x556b7e4f70e0 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26877fe640 1 -- 172.21.15.38:0/3152509194 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26980ccb50 con 0x556b7e4f70e0 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26b11bb740 10 client.15096 fetch_fsmap learned FSMap version 22 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.634+0000 7f26b11bb740 1 -- 172.21.15.38:0/3152509194 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x556b7e565250 con 0x556b7e4f70e0 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.635+0000 7f26877fe640 1 -- 172.21.15.38:0/3152509194 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26a0052070 con 0x556b7e4f70e0 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.635+0000 7f26877fe640 1 -- 172.21.15.38:0/3152509194 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f26980afb70 con 0x556b7e4f70e0 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.635+0000 7f26b11bb740 10 client.15096 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.635+0000 7f26b11bb740 10 client.15096 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.635+0000 7f26b11bb740 20 client.15096 populate_metadata read hostname 'smithi038' 2024-03-26T06:36:54.644 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.635+0000 7f26b11bb740 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556b7e569460 0x556b7e565420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.635+0000 7f26b11bb740 4 client.15096 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.635+0000 7f26b11bb740 1 -- 172.21.15.38:0/3152509194 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556b7e464050 con 0x556b7e569460 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.635+0000 7f26a77fe640 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556b7e569460 0x556b7e565420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.635+0000 7f26a77fe640 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556b7e569460 0x556b7e565420 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edd786c40a4706c2 server_cookie=b496886f3b574529 in_seq=0 out_seq=0 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.635+0000 7f26877fe640 10 client.15096 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26877fe640 1 -- 172.21.15.38:0/3152509194 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x556b7e464050 con 0x556b7e569460 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26877fe640 10 client.15096 handle_command_reply: tid=0 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 1 client.15096 shutdown 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 1 -- 172.21.15.38:0/3152509194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 msgr2=0x556b7e505760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e505760 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f26980058c0 tx=0x7f2698097b60 comp rx=0 tx=0).stop 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 1 -- 172.21.15.38:0/3152509194 shutdown_connections 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556b7e569460 0x556b7e565420 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556b7e4fcc40 0x556b7e563cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.645 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556b7e4f7ad0 0x556b7e505ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26877fe640 0 client.15096 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:36:54.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 1 --2- 172.21.15.38:0/3152509194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556b7e4f70e0 0x556b7e505760 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 1 -- 172.21.15.38:0/3152509194 >> 172.21.15.38:0/3152509194 conn(0x556b7e4bf400 msgr2=0x556b7e4f8ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 1 -- 172.21.15.38:0/3152509194 shutdown_connections 2024-03-26T06:36:54.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 1 -- 172.21.15.38:0/3152509194 wait complete. 2024-03-26T06:36:54.646 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26b11bb740 20 client.15096 trim_cache size 0 max 16384 2024-03-26T06:36:54.646 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:36:54.646 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c2824556-eb39-11ee-95d0-87774f69a715", 2024-03-26T06:36:54.646 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 0, 2024-03-26T06:36:54.647 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24539, 2024-03-26T06:36:54.647 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-03-26T06:36:54.647 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-03-26T06:36:54.647 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-03-26T06:36:54.647 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 87.327790246999996, 2024-03-26T06:36:54.647 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 22, 2024-03-26T06:36:54.647 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 88, 2024-03-26T06:36:54.647 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 86, 2024-03-26T06:36:54.647 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 92.765535297 2024-03-26T06:36:54.647 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:36:54.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26ae4f8640 1 -- 172.21.15.38:0/3231927381 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f26900a2890 msgr2=0x7f26900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26ae4f8640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f26900a2890 0x7f26900a4d50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f26a0005e30 tx=0x7f26a002cca0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26ae4f8640 1 -- 172.21.15.38:0/3231927381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a8101300 msgr2=0x7f26a8197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:54.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.636+0000 7f26ae4f8640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a8101300 0x7f26a8197530 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f269800e9c0 tx=0x7f269800edc0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.637+0000 7f26ae4f8640 1 -- 172.21.15.38:0/3231927381 shutdown_connections 2024-03-26T06:36:54.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.637+0000 7f26ae4f8640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f26900a2890 0x7f26900a4d50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.648 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.637+0000 7f26ae4f8640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f26a8102eb0 0x7f26a819eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.637+0000 7f26ae4f8640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f26a81024f0 0x7f26a8197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.637+0000 7f26ae4f8640 1 --2- 172.21.15.38:0/3231927381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f26a8101300 0x7f26a8197530 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.637+0000 7f26ae4f8640 1 -- 172.21.15.38:0/3231927381 >> 172.21.15.38:0/3231927381 conn(0x7f26a80fc890 msgr2=0x7f26a8105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.637+0000 7f26ae4f8640 1 -- 172.21.15.38:0/3231927381 shutdown_connections 2024-03-26T06:36:54.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:54.637+0000 7f26ae4f8640 1 -- 172.21.15.38:0/3231927381 wait complete. 2024-03-26T06:36:54.650 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:54 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1459136603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:36:54.650 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:54 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/258703971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:36:54.650 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:54 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3183548765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:36:54.650 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T06:36:54.651 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T06:36:54.661 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T06:36:54.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- bash -c 'ceph orch status' 2024-03-26T06:36:54.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:54 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1459136603' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:36:54.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:54 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/258703971' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:36:54.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:54 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3183548765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:36:54.904 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:55.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:55 smithi121 ceph-mon[36349]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:55.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:55 smithi170 ceph-mon[36912]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:55.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:55 smithi038 ceph-mon[32265]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:36:56.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.576+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/4038216385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f68101a20 msgr2=0x7f0f68101ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:56.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.576+0000 7f0f6dc6f700 1 --2- 172.21.15.38:0/4038216385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f68101a20 0x7f0f68101ea0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f0f60009990 tx=0x7f0f60009ca0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.576+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/4038216385 shutdown_connections 2024-03-26T06:36:56.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.576+0000 7f0f6dc6f700 1 --2- 172.21.15.38:0/4038216385 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f68101a20 0x7f0f68101ea0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.576+0000 7f0f6dc6f700 1 --2- 172.21.15.38:0/4038216385 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f68100700 0x7f0f68100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.576+0000 7f0f6dc6f700 1 --2- 172.21.15.38:0/4038216385 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f68101060 0x7f0f681014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.576+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/4038216385 >> 172.21.15.38:0/4038216385 conn(0x7f0f680fc2a0 msgr2=0x7f0f680fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:56.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.576+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/4038216385 shutdown_connections 2024-03-26T06:36:56.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.576+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/4038216385 wait complete. 2024-03-26T06:36:56.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f6dc6f700 1 Processor -- start 2024-03-26T06:36:56.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f6dc6f700 1 -- start start 2024-03-26T06:36:56.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f6dc6f700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f68100700 0x7f0f6819a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:56.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f6dc6f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f68101060 0x7f0f6819a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:56.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f6dc6f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f68101a20 0x7f0f681a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f6dc6f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6810fbf0 con 0x7f0f68101a20 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f6dc6f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6810fa70 con 0x7f0f68101060 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f6dc6f700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f6810fd70 con 0x7f0f68100700 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f67fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f68101a20 0x7f0f681a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f67fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f68101a20 0x7f0f681a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58168/0 (socket says 172.21.15.38:58168) 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f67fff700 1 -- 172.21.15.38:0/2941040707 learned_addr learned my addr 172.21.15.38:0/2941040707 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f677fe700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f68100700 0x7f0f6819a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f66ffd700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f68101060 0x7f0f6819a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f67fff700 1 -- 172.21.15.38:0/2941040707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f68100700 msgr2=0x7f0f6819a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f67fff700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f68100700 0x7f0f6819a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f67fff700 1 -- 172.21.15.38:0/2941040707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f68101060 msgr2=0x7f0f6819a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.577+0000 7f0f67fff700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f68101060 0x7f0f6819a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.578+0000 7f0f67fff700 1 -- 172.21.15.38:0/2941040707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f681a1da0 con 0x7f0f68101a20 2024-03-26T06:36:56.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.578+0000 7f0f67fff700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f68101a20 0x7f0f681a1700 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0f60006010 tx=0x7f0f600047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:56.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.578+0000 7f0f64ff9700 1 -- 172.21.15.38:0/2941040707 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60003c50 con 0x7f0f68101a20 2024-03-26T06:36:56.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.578+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/2941040707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f681a2030 con 0x7f0f68101a20 2024-03-26T06:36:56.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.578+0000 7f0f64ff9700 1 -- 172.21.15.38:0/2941040707 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f60004830 con 0x7f0f68101a20 2024-03-26T06:36:56.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.578+0000 7f0f64ff9700 1 -- 172.21.15.38:0/2941040707 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f600053e0 con 0x7f0f68101a20 2024-03-26T06:36:56.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.578+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/2941040707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f681a2480 con 0x7f0f68101a20 2024-03-26T06:36:56.583 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.579+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/2941040707 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f68068aa0 con 0x7f0f68101a20 2024-03-26T06:36:56.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.580+0000 7f0f64ff9700 1 -- 172.21.15.38:0/2941040707 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0f60005580 con 0x7f0f68101a20 2024-03-26T06:36:56.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.580+0000 7f0f64ff9700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0f540a2920 0x7f0f540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:56.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.581+0000 7f0f677fe700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0f540a2920 0x7f0f540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:56.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.581+0000 7f0f64ff9700 1 -- 172.21.15.38:0/2941040707 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f0f600cfb60 con 0x7f0f68101a20 2024-03-26T06:36:56.584 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.581+0000 7f0f677fe700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0f540a2920 0x7f0f540a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009550 tx=0x7f0f580091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:56.587 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.585+0000 7f0f64ff9700 1 -- 172.21.15.38:0/2941040707 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0f60013070 con 0x7f0f68101a20 2024-03-26T06:36:56.810 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.807+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/2941040707 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f68062220 con 0x7f0f540a2920 2024-03-26T06:36:56.811 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.808+0000 7f0f64ff9700 1 -- 172.21.15.38:0/2941040707 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0f68062220 con 0x7f0f540a2920 2024-03-26T06:36:56.811 INFO:teuthology.orchestra.run.smithi038.stdout:Backend: cephadm 2024-03-26T06:36:56.811 INFO:teuthology.orchestra.run.smithi038.stdout:Available: Yes 2024-03-26T06:36:56.811 INFO:teuthology.orchestra.run.smithi038.stdout:Paused: No 2024-03-26T06:36:56.813 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/2941040707 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0f540a2920 msgr2=0x7f0f540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:56.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0f540a2920 0x7f0f540a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009550 tx=0x7f0f580091b0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/2941040707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f68101a20 msgr2=0x7f0f681a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:56.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f68101a20 0x7f0f681a1700 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0f60006010 tx=0x7f0f600047b0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/2941040707 shutdown_connections 2024-03-26T06:36:56.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f68101a20 0x7f0f681a1700 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0f68100700 0x7f0f6819a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f0f540a2920 0x7f0f540a4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 --2- 172.21.15.38:0/2941040707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f68101060 0x7f0f6819a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:56.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/2941040707 >> 172.21.15.38:0/2941040707 conn(0x7f0f680fc2a0 msgr2=0x7f0f680fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:56.814 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/2941040707 shutdown_connections 2024-03-26T06:36:56.815 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:56.811+0000 7f0f6dc6f700 1 -- 172.21.15.38:0/2941040707 wait complete. 2024-03-26T06:36:57.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:57 smithi038 ceph-mon[32265]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-26T06:36:57.488 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:57 smithi038 ceph-mon[32265]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:36:57.517 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-26T06:36:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:57 smithi121 ceph-mon[36349]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-26T06:36:57.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:57 smithi121 ceph-mon[36349]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:36:57.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:57 smithi170 ceph-mon[36912]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-03-26T06:36:57.671 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:57 smithi170 ceph-mon[36912]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:36:57.771 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:36:58.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:58 smithi038 ceph-mon[32265]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-26T06:36:58.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:58 smithi121 ceph-mon[36349]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-26T06:36:58.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:58 smithi170 ceph-mon[36912]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-03-26T06:36:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.484+0000 7f94e9017700 1 -- 172.21.15.38:0/525218899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e4100d10 msgr2=0x7f94e4101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.484+0000 7f94e9017700 1 --2- 172.21.15.38:0/525218899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e4100d10 0x7f94e4101130 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009990 tx=0x7f94d4009ca0 comp rx=0 tx=0).stop 2024-03-26T06:36:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.484+0000 7f94e9017700 1 -- 172.21.15.38:0/525218899 shutdown_connections 2024-03-26T06:36:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.484+0000 7f94e9017700 1 --2- 172.21.15.38:0/525218899 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94e41028b0 0x7f94e4109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.484+0000 7f94e9017700 1 --2- 172.21.15.38:0/525218899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e4100d10 0x7f94e4101130 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.484+0000 7f94e9017700 1 --2- 172.21.15.38:0/525218899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94e4101f10 0x7f94e4102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.484+0000 7f94e9017700 1 -- 172.21.15.38:0/525218899 >> 172.21.15.38:0/525218899 conn(0x7f94e40fc2b0 msgr2=0x7f94e40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.484+0000 7f94e9017700 1 -- 172.21.15.38:0/525218899 shutdown_connections 2024-03-26T06:36:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.484+0000 7f94e9017700 1 -- 172.21.15.38:0/525218899 wait complete. 2024-03-26T06:36:59.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e9017700 1 Processor -- start 2024-03-26T06:36:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e9017700 1 -- start start 2024-03-26T06:36:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e9017700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94e4100d10 0x7f94e4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e9017700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e4101f10 0x7f94e41985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e9017700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94e41028b0 0x7f94e419f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e9017700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e410d9e0 con 0x7f94e4101f10 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e9017700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e410d860 con 0x7f94e41028b0 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e9017700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e410db60 con 0x7f94e4100d10 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e4101f10 0x7f94e41985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e4101f10 0x7f94e41985d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58178/0 (socket says 172.21.15.38:58178) 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e259c700 1 -- 172.21.15.38:0/3846909128 learned_addr learned my addr 172.21.15.38:0/3846909128 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e2d9d700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94e4100d10 0x7f94e4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e359e700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94e41028b0 0x7f94e419f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e259c700 1 -- 172.21.15.38:0/3846909128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94e4100d10 msgr2=0x7f94e4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e259c700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94e4100d10 0x7f94e4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e259c700 1 -- 172.21.15.38:0/3846909128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94e41028b0 msgr2=0x7f94e419f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e259c700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94e41028b0 0x7f94e419f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.485+0000 7f94e259c700 1 -- 172.21.15.38:0/3846909128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e419fcf0 con 0x7f94e4101f10 2024-03-26T06:36:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.486+0000 7f94e259c700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e4101f10 0x7f94e41985d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00d980 tx=0x7f94cc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.486+0000 7f94dbfff700 1 -- 172.21.15.38:0/3846909128 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc016070 con 0x7f94e4101f10 2024-03-26T06:36:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.486+0000 7f94e9017700 1 -- 172.21.15.38:0/3846909128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e419ffe0 con 0x7f94e4101f10 2024-03-26T06:36:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.486+0000 7f94dbfff700 1 -- 172.21.15.38:0/3846909128 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94cc009db0 con 0x7f94e4101f10 2024-03-26T06:36:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.486+0000 7f94dbfff700 1 -- 172.21.15.38:0/3846909128 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94cc00f3c0 con 0x7f94e4101f10 2024-03-26T06:36:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.486+0000 7f94e9017700 1 -- 172.21.15.38:0/3846909128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e41a04e0 con 0x7f94e4101f10 2024-03-26T06:36:59.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.487+0000 7f94e9017700 1 -- 172.21.15.38:0/3846909128 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e4068aa0 con 0x7f94e4101f10 2024-03-26T06:36:59.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.488+0000 7f94dbfff700 1 -- 172.21.15.38:0/3846909128 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f94cc019070 con 0x7f94e4101f10 2024-03-26T06:36:59.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.488+0000 7f94dbfff700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f94d00a2890 0x7f94d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:36:59.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.489+0000 7f94e2d9d700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f94d00a2890 0x7f94d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:36:59.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.489+0000 7f94dbfff700 1 -- 172.21.15.38:0/3846909128 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f94cc0ce4f0 con 0x7f94e4101f10 2024-03-26T06:36:59.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.489+0000 7f94e2d9d700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f94d00a2890 0x7f94d00a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f94d400b5c0 tx=0x7f94d4003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:36:59.495 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.493+0000 7f94dbfff700 1 -- 172.21.15.38:0/3846909128 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f94cc09b590 con 0x7f94e4101f10 2024-03-26T06:36:59.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:36:59 smithi121 ceph-mon[36349]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:36:59.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:36:59 smithi170 ceph-mon[36912]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:36:59.709 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.706+0000 7f94e9017700 1 -- 172.21.15.38:0/3846909128 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f94e4062220 con 0x7f94d00a2890 2024-03-26T06:36:59.709 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:36:59 smithi038 ceph-mon[32265]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:36:59.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.722+0000 7f94dbfff700 1 -- 172.21.15.38:0/3846909128 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f94e4062220 con 0x7f94d00a2890 2024-03-26T06:36:59.725 INFO:teuthology.orchestra.run.smithi038.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T06:36:59.725 INFO:teuthology.orchestra.run.smithi038.stdout:mds.a smithi038 running (101s) 17s ago 100s 16.0M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 7fdcfc8c1d62 2024-03-26T06:36:59.725 INFO:teuthology.orchestra.run.smithi038.stdout:mds.b smithi121 running (98s) 31s ago 97s 24.8M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 6b23d6d3808e 2024-03-26T06:36:59.725 INFO:teuthology.orchestra.run.smithi038.stdout:mds.c smithi170 running (112s) 31s ago 111s 17.9M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 33d915e9dd0a 2024-03-26T06:36:59.725 INFO:teuthology.orchestra.run.smithi038.stdout:mds.d smithi038 running (2m) 17s ago 119s 16.4M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 35bfa5bb4a68 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mds.e smithi121 running (117s) 31s ago 117s 18.1M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 1fd8477c68b1 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mds.f smithi170 running (114s) 31s ago 114s 15.9M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 272d2c6aaa37 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mds.g smithi038 running (103s) 17s ago 103s 16.0M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 007d8530a19e 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mds.h smithi121 running (109s) 31s ago 109s 16.4M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 d6d6c8cc8bc5 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mds.i smithi170 running (106s) 31s ago 106s 19.9M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 fea61e049736 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mds.j smithi038 running (2m) 17s ago 2m 13.0M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 386c31cfabf7 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mds.k smithi121 running (2m) 31s ago 2m 13.9M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 15dddf100d4a 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mds.l smithi170 running (2m) 31s ago 2m 14.0M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 f2e56bf0d4bd 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.x smithi038 *:9283,8765 running (9m) 17s ago 9m 512M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 1c39329e522c 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.y smithi121 *:8443,8765 running (7m) 31s ago 7m 453M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 bcb10b978ad0 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.z smithi170 *:8443,8765 running (7m) 31s ago 7m 450M - 19.0.0-1964-g7e98f5f6 10cdb6adb615 5062b4749cc8 2024-03-26T06:36:59.726 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a smithi038 running (9m) 17s ago 9m 49.4M 2048M 19.0.0-1964-g7e98f5f6 10cdb6adb615 127b2bec7e81 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b smithi121 running (7m) 31s ago 7m 37.5M 2048M 19.0.0-1964-g7e98f5f6 10cdb6adb615 015bdab4d436 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c smithi170 running (7m) 31s ago 7m 41.4M 2048M 19.0.0-1964-g7e98f5f6 10cdb6adb615 bceeb49f7778 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 smithi038 running (7m) 17s ago 7m 50.7M 4273M 19.0.0-1964-g7e98f5f6 10cdb6adb615 3711054aa635 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 smithi121 running (6m) 31s ago 6m 48.7M 4270M 19.0.0-1964-g7e98f5f6 10cdb6adb615 f841e6ca2e29 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 smithi170 running (6m) 31s ago 6m 56.2M 4270M 19.0.0-1964-g7e98f5f6 10cdb6adb615 057b6e93c12b 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 smithi038 running (5m) 17s ago 5m 57.1M 4273M 19.0.0-1964-g7e98f5f6 10cdb6adb615 6a7c93c75e73 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 smithi121 running (5m) 31s ago 5m 45.7M 4270M 19.0.0-1964-g7e98f5f6 10cdb6adb615 3b8039a2d284 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 smithi170 running (5m) 31s ago 5m 49.4M 4270M 19.0.0-1964-g7e98f5f6 10cdb6adb615 a3be616570bc 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 smithi038 running (4m) 17s ago 4m 52.6M 4273M 19.0.0-1964-g7e98f5f6 10cdb6adb615 5f6b68aaa1b8 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 smithi121 running (4m) 31s ago 4m 53.0M 4270M 19.0.0-1964-g7e98f5f6 10cdb6adb615 cadb09d084a8 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8 smithi170 running (3m) 31s ago 3m 54.2M 4270M 19.0.0-1964-g7e98f5f6 10cdb6adb615 c9464b613a64 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9 smithi038 running (3m) 17s ago 3m 47.0M 4273M 19.0.0-1964-g7e98f5f6 10cdb6adb615 7f96f26d1113 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10 smithi121 running (2m) 31s ago 2m 49.2M 4270M 19.0.0-1964-g7e98f5f6 10cdb6adb615 88d06d00d892 2024-03-26T06:36:59.727 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11 smithi170 running (2m) 31s ago 2m 43.5M 4270M 19.0.0-1964-g7e98f5f6 10cdb6adb615 51a5b8cfc935 2024-03-26T06:36:59.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 -- 172.21.15.38:0/3846909128 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f94d00a2890 msgr2=0x7f94d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:59.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f94d00a2890 0x7f94d00a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f94d400b5c0 tx=0x7f94d4003960 comp rx=0 tx=0).stop 2024-03-26T06:36:59.728 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 -- 172.21.15.38:0/3846909128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e4101f10 msgr2=0x7f94e41985d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:36:59.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e4101f10 0x7f94e41985d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f94cc00d980 tx=0x7f94cc00dc90 comp rx=0 tx=0).stop 2024-03-26T06:36:59.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 -- 172.21.15.38:0/3846909128 shutdown_connections 2024-03-26T06:36:59.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f94d00a2890 0x7f94d00a4d50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:59.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f94e41028b0 0x7f94e419f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:59.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f94e4100d10 0x7f94e4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:59.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 --2- 172.21.15.38:0/3846909128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f94e4101f10 0x7f94e41985d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:36:59.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 -- 172.21.15.38:0/3846909128 >> 172.21.15.38:0/3846909128 conn(0x7f94e40fc2b0 msgr2=0x7f94e4105330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:36:59.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 -- 172.21.15.38:0/3846909128 shutdown_connections 2024-03-26T06:36:59.729 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:36:59.725+0000 7f94e9017700 1 -- 172.21.15.38:0/3846909128 wait complete. 2024-03-26T06:37:00.391 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-26T06:37:00.643 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:37:00.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:00 smithi121 ceph-mon[36349]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:00.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:00 smithi170 ceph-mon[36912]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:00.763 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:00 smithi038 ceph-mon[32265]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:01.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:01 smithi121 ceph-mon[36349]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:37:01.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:01 smithi170 ceph-mon[36912]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:37:01.711 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:01 smithi038 ceph-mon[32265]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:37:02.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.275+0000 7f08986f5700 1 -- 172.21.15.38:0/3237308086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0890102890 msgr2=0x7f0890104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:02.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.275+0000 7f08986f5700 1 --2- 172.21.15.38:0/3237308086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0890102890 0x7f0890104c80 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f088c009990 tx=0x7f088c009ca0 comp rx=0 tx=0).stop 2024-03-26T06:37:02.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.275+0000 7f08986f5700 1 -- 172.21.15.38:0/3237308086 shutdown_connections 2024-03-26T06:37:02.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 --2- 172.21.15.38:0/3237308086 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0890107af0 0x7f0890109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:02.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 --2- 172.21.15.38:0/3237308086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0890102890 0x7f0890104c80 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:02.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 --2- 172.21.15.38:0/3237308086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f08901051c0 0x7f08901075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:02.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 -- 172.21.15.38:0/3237308086 >> 172.21.15.38:0/3237308086 conn(0x7f08900fc2c0 msgr2=0x7f08900fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:02.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 -- 172.21.15.38:0/3237308086 shutdown_connections 2024-03-26T06:37:02.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 -- 172.21.15.38:0/3237308086 wait complete. 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 Processor -- start 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 -- start start 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0890102890 0x7f0890198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08901051c0 0x7f08901985d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0890107af0 0x7f089019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089010da50 con 0x7f0890107af0 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089010d8d0 con 0x7f0890102890 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f08986f5700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089010dbd0 con 0x7f08901051c0 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.276+0000 7f0896c92700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0890107af0 0x7f089019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f0896c92700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0890107af0 0x7f089019f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:58198/0 (socket says 172.21.15.38:58198) 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f0896c92700 1 -- 172.21.15.38:0/2216193292 learned_addr learned my addr 172.21.15.38:0/2216193292 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:02.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f0896491700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0890102890 0x7f0890198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f0895c90700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08901051c0 0x7f08901985d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f0896c92700 1 -- 172.21.15.38:0/2216193292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08901051c0 msgr2=0x7f08901985d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f0896c92700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08901051c0 0x7f08901985d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f0896c92700 1 -- 172.21.15.38:0/2216193292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0890102890 msgr2=0x7f0890198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f0896c92700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0890102890 0x7f0890198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f0896c92700 1 -- 172.21.15.38:0/2216193292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089019fca0 con 0x7f0890107af0 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f0896c92700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0890107af0 0x7f089019f600 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f088800ba10 tx=0x7f088800bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f08877fe700 1 -- 172.21.15.38:0/2216193292 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888019070 con 0x7f0890107af0 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f08877fe700 1 -- 172.21.15.38:0/2216193292 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f088800ce10 con 0x7f0890107af0 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f08986f5700 1 -- 172.21.15.38:0/2216193292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f089019ff90 con 0x7f0890107af0 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f08877fe700 1 -- 172.21.15.38:0/2216193292 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888012340 con 0x7f0890107af0 2024-03-26T06:37:02.300 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.277+0000 7f08986f5700 1 -- 172.21.15.38:0/2216193292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08901a04c0 con 0x7f0890107af0 2024-03-26T06:37:02.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.278+0000 7f08986f5700 1 -- 172.21.15.38:0/2216193292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0890068aa0 con 0x7f0890107af0 2024-03-26T06:37:02.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.279+0000 7f08877fe700 1 -- 172.21.15.38:0/2216193292 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f088801c070 con 0x7f0890107af0 2024-03-26T06:37:02.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.280+0000 7f08877fe700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f087c0a2910 0x7f087c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:02.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.280+0000 7f0896491700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f087c0a2910 0x7f087c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:02.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.280+0000 7f08877fe700 1 -- 172.21.15.38:0/2216193292 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f08880ce190 con 0x7f0890107af0 2024-03-26T06:37:02.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.281+0000 7f0896491700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f087c0a2910 0x7f087c0a4dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f088c00b5c0 tx=0x7f088c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:02.301 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.284+0000 7f08877fe700 1 -- 172.21.15.38:0/2216193292 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f088809b1f0 con 0x7f0890107af0 2024-03-26T06:37:02.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.497+0000 7f08986f5700 1 -- 172.21.15.38:0/2216193292 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0890062220 con 0x7f087c0a2910 2024-03-26T06:37:02.505 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.503+0000 7f08877fe700 1 -- 172.21.15.38:0/2216193292 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f0890062220 con 0x7f087c0a2910 2024-03-26T06:37:02.505 INFO:teuthology.orchestra.run.smithi038.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-26T06:37:02.506 INFO:teuthology.orchestra.run.smithi038.stdout:mds.all 12/12 34s ago 2m smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi121=b;smithi121=e;smithi121=h;smithi121=k;smithi170=c;smithi170=f;smithi170=i;smithi170=l;count:12 2024-03-26T06:37:02.506 INFO:teuthology.orchestra.run.smithi038.stdout:mgr 3/3 34s ago 7m smithi038=x;smithi121=y;smithi170=z;count:3 2024-03-26T06:37:02.506 INFO:teuthology.orchestra.run.smithi038.stdout:mon 3/3 34s ago 7m smithi038:172.21.15.38=a;smithi121:172.21.15.121=b;smithi170:172.21.15.170=c;count:3 2024-03-26T06:37:02.506 INFO:teuthology.orchestra.run.smithi038.stdout:osd 12 34s ago - 2024-03-26T06:37:02.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 -- 172.21.15.38:0/2216193292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f087c0a2910 msgr2=0x7f087c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:02.507 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f087c0a2910 0x7f087c0a4dd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f088c00b5c0 tx=0x7f088c005fd0 comp rx=0 tx=0).stop 2024-03-26T06:37:02.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 -- 172.21.15.38:0/2216193292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0890107af0 msgr2=0x7f089019f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:02.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0890107af0 0x7f089019f600 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f088800ba10 tx=0x7f088800bd20 comp rx=0 tx=0).stop 2024-03-26T06:37:02.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 -- 172.21.15.38:0/2216193292 shutdown_connections 2024-03-26T06:37:02.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0890107af0 0x7f089019f600 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:02.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f087c0a2910 0x7f087c0a4dd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:02.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0890102890 0x7f0890198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:02.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 --2- 172.21.15.38:0/2216193292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f08901051c0 0x7f08901985d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:02.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 -- 172.21.15.38:0/2216193292 >> 172.21.15.38:0/2216193292 conn(0x7f08900fc2c0 msgr2=0x7f08901031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:02.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 -- 172.21.15.38:0/2216193292 shutdown_connections 2024-03-26T06:37:02.508 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:02.505+0000 7f08986f5700 1 -- 172.21.15.38:0/2216193292 wait complete. 2024-03-26T06:37:03.267 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-26T06:37:03.516 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:37:03.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:03 smithi038 ceph-mon[32265]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-26T06:37:03.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:03 smithi038 ceph-mon[32265]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:03.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:03 smithi121 ceph-mon[36349]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-26T06:37:03.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:03 smithi121 ceph-mon[36349]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:03.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:03 smithi170 ceph-mon[36912]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-26T06:37:03.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:03 smithi170 ceph-mon[36912]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:05.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.201+0000 7fcb12e50700 1 -- 172.21.15.38:0/1083619821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb0c074840 msgr2=0x7fcb0c074cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.201+0000 7fcb12e50700 1 --2- 172.21.15.38:0/1083619821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb0c074840 0x7fcb0c074cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8009990 tx=0x7fcaf8009ca0 comp rx=0 tx=0).stop 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 -- 172.21.15.38:0/1083619821 shutdown_connections 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 --2- 172.21.15.38:0/1083619821 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb0c10e3f0 0x7fcb0c1107e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 --2- 172.21.15.38:0/1083619821 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb0c075e10 0x7fcb0c074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 --2- 172.21.15.38:0/1083619821 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb0c074840 0x7fcb0c074cc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 -- 172.21.15.38:0/1083619821 >> 172.21.15.38:0/1083619821 conn(0x7fcb0c0fd3f0 msgr2=0x7fcb0c0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 -- 172.21.15.38:0/1083619821 shutdown_connections 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 -- 172.21.15.38:0/1083619821 wait complete. 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 Processor -- start 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 -- start start 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb0c074840 0x7fcb0c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb0c075e10 0x7fcb0c19ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb0c10e3f0 0x7fcb0c1a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c114070 con 0x7fcb0c074840 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c113ef0 con 0x7fcb0c10e3f0 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.202+0000 7fcb12e50700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb0c1141f0 con 0x7fcb0c075e10 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb10bec700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb0c074840 0x7fcb0c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb10bec700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb0c074840 0x7fcb0c19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43704/0 (socket says 172.21.15.38:43704) 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb10bec700 1 -- 172.21.15.38:0/918212497 learned_addr learned my addr 172.21.15.38:0/918212497 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb0bfff700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb0c075e10 0x7fcb0c19ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb113ed700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb0c10e3f0 0x7fcb0c1a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb10bec700 1 -- 172.21.15.38:0/918212497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb0c075e10 msgr2=0x7fcb0c19ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb10bec700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb0c075e10 0x7fcb0c19ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb10bec700 1 -- 172.21.15.38:0/918212497 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb0c10e3f0 msgr2=0x7fcb0c1a5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb10bec700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb0c10e3f0 0x7fcb0c1a5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb10bec700 1 -- 172.21.15.38:0/918212497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb0c1a62d0 con 0x7fcb0c074840 2024-03-26T06:37:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb10bec700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb0c074840 0x7fcb0c19e670 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000b530 tx=0x7fcb0000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb09ffb700 1 -- 172.21.15.38:0/918212497 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00015020 con 0x7fcb0c074840 2024-03-26T06:37:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb09ffb700 1 -- 172.21.15.38:0/918212497 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb00011d10 con 0x7fcb0c074840 2024-03-26T06:37:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb12e50700 1 -- 172.21.15.38:0/918212497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb0c1a65c0 con 0x7fcb0c074840 2024-03-26T06:37:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.203+0000 7fcb09ffb700 1 -- 172.21.15.38:0/918212497 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb00010d70 con 0x7fcb0c074840 2024-03-26T06:37:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.204+0000 7fcb12e50700 1 -- 172.21.15.38:0/918212497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb0c1a6bd0 con 0x7fcb0c074840 2024-03-26T06:37:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.204+0000 7fcb12e50700 1 -- 172.21.15.38:0/918212497 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0c068aa0 con 0x7fcb0c074840 2024-03-26T06:37:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.206+0000 7fcb09ffb700 1 -- 172.21.15.38:0/918212497 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcb00018020 con 0x7fcb0c074840 2024-03-26T06:37:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.206+0000 7fcb09ffb700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcafc0a2910 0x7fcafc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.206+0000 7fcb0bfff700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcafc0a2910 0x7fcafc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.206+0000 7fcb09ffb700 1 -- 172.21.15.38:0/918212497 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcb000ce620 con 0x7fcb0c074840 2024-03-26T06:37:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.207+0000 7fcb0bfff700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcafc0a2910 0x7fcafc0a4dd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8003890 tx=0x7fcaf8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:05.212 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.210+0000 7fcb09ffb700 1 -- 172.21.15.38:0/918212497 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcb0001f020 con 0x7fcb0c074840 2024-03-26T06:37:05.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.419+0000 7fcb12e50700 1 -- 172.21.15.38:0/918212497 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb0c062220 con 0x7fcafc0a2910 2024-03-26T06:37:05.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.421+0000 7fcb09ffb700 1 -- 172.21.15.38:0/918212497 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fcb0c062220 con 0x7fcafc0a2910 2024-03-26T06:37:05.423 INFO:teuthology.orchestra.run.smithi038.stdout:HOST ADDR LABELS STATUS 2024-03-26T06:37:05.423 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 172.21.15.38 2024-03-26T06:37:05.424 INFO:teuthology.orchestra.run.smithi038.stdout:smithi121 172.21.15.121 2024-03-26T06:37:05.424 INFO:teuthology.orchestra.run.smithi038.stdout:smithi170 172.21.15.170 2024-03-26T06:37:05.424 INFO:teuthology.orchestra.run.smithi038.stdout:3 hosts in cluster 2024-03-26T06:37:05.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 -- 172.21.15.38:0/918212497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcafc0a2910 msgr2=0x7fcafc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:05.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcafc0a2910 0x7fcafc0a4dd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8003890 tx=0x7fcaf8005fd0 comp rx=0 tx=0).stop 2024-03-26T06:37:05.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 -- 172.21.15.38:0/918212497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb0c074840 msgr2=0x7fcb0c19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:05.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb0c074840 0x7fcb0c19e670 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcb0000b530 tx=0x7fcb0000b840 comp rx=0 tx=0).stop 2024-03-26T06:37:05.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 -- 172.21.15.38:0/918212497 shutdown_connections 2024-03-26T06:37:05.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcafc0a2910 0x7fcafc0a4dd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:05.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb0c10e3f0 0x7fcb0c1a5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:05.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb0c074840 0x7fcb0c19e670 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:05.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 --2- 172.21.15.38:0/918212497 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcb0c075e10 0x7fcb0c19ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:05.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 -- 172.21.15.38:0/918212497 >> 172.21.15.38:0/918212497 conn(0x7fcb0c0fd3f0 msgr2=0x7fcb0c101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:05.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 -- 172.21.15.38:0/918212497 shutdown_connections 2024-03-26T06:37:05.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:05.423+0000 7fcb12e50700 1 -- 172.21.15.38:0/918212497 wait complete. 2024-03-26T06:37:05.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:05 smithi121 ceph-mon[36349]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-26T06:37:05.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:05 smithi170 ceph-mon[36912]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-26T06:37:05.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:05 smithi038 ceph-mon[32265]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-03-26T06:37:06.155 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-26T06:37:06.401 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:37:06.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:06 smithi170 ceph-mon[36912]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:06.721 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:06 smithi038 ceph-mon[32265]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:06.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:06 smithi121 ceph-mon[36349]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:07.542 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:07 smithi038 ceph-mon[32265]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-26T06:37:07.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:07 smithi170 ceph-mon[36912]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-26T06:37:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:07 smithi121 ceph-mon[36349]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-26T06:37:08.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.197+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/1524661685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c88107b00 msgr2=0x7f1c88109f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:08.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.197+0000 7f1c8e3c5700 1 --2- 172.21.15.38:0/1524661685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c88107b00 0x7f1c88109f70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000b0a0 tx=0x7f1c8000b3b0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.197+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/1524661685 shutdown_connections 2024-03-26T06:37:08.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.197+0000 7f1c8e3c5700 1 --2- 172.21.15.38:0/1524661685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c88107b00 0x7f1c88109f70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.197+0000 7f1c8e3c5700 1 --2- 172.21.15.38:0/1524661685 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c881028a0 0x7f1c88104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.197+0000 7f1c8e3c5700 1 --2- 172.21.15.38:0/1524661685 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c881051d0 0x7f1c881075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.197+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/1524661685 >> 172.21.15.38:0/1524661685 conn(0x7f1c880fc2b0 msgr2=0x7f1c880fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:08.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.197+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/1524661685 shutdown_connections 2024-03-26T06:37:08.200 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.197+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/1524661685 wait complete. 2024-03-26T06:37:08.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.197+0000 7f1c8e3c5700 1 Processor -- start 2024-03-26T06:37:08.201 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.197+0000 7f1c8e3c5700 1 -- start start 2024-03-26T06:37:08.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8e3c5700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c881028a0 0x7f1c88102200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:08.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8e3c5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c881051d0 0x7f1c88100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:08.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8e3c5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c88107b00 0x7f1c88100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:08.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8e3c5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8810d9f0 con 0x7f1c88107b00 2024-03-26T06:37:08.202 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8e3c5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8810d870 con 0x7f1c881051d0 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8e3c5700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8810db70 con 0x7f1c881028a0 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8c962700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c88107b00 0x7f1c88100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8c962700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c88107b00 0x7f1c88100d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43718/0 (socket says 172.21.15.38:43718) 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8c962700 1 -- 172.21.15.38:0/2575295303 learned_addr learned my addr 172.21.15.38:0/2575295303 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c87fff700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c881028a0 0x7f1c88102200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c877fe700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c881051d0 0x7f1c88100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8c962700 1 -- 172.21.15.38:0/2575295303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c881028a0 msgr2=0x7f1c88102200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8c962700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c881028a0 0x7f1c88102200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8c962700 1 -- 172.21.15.38:0/2575295303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c881051d0 msgr2=0x7f1c88100850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8c962700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c881051d0 0x7f1c88100850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8c962700 1 -- 172.21.15.38:0/2575295303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c88101640 con 0x7f1c88107b00 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c87fff700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c881028a0 0x7f1c88102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:37:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c8c962700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c88107b00 0x7f1c88100d90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000b070 tx=0x7f1c80004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.198+0000 7f1c857fa700 1 -- 172.21.15.38:0/2575295303 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80021070 con 0x7f1c88107b00 2024-03-26T06:37:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.199+0000 7f1c857fa700 1 -- 172.21.15.38:0/2575295303 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c80007c80 con 0x7f1c88107b00 2024-03-26T06:37:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.199+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/2575295303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c881a2160 con 0x7f1c88107b00 2024-03-26T06:37:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.199+0000 7f1c857fa700 1 -- 172.21.15.38:0/2575295303 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80005430 con 0x7f1c88107b00 2024-03-26T06:37:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.199+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/2575295303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c881a25c0 con 0x7f1c88107b00 2024-03-26T06:37:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.200+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/2575295303 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c88068aa0 con 0x7f1c88107b00 2024-03-26T06:37:08.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.201+0000 7f1c857fa700 1 -- 172.21.15.38:0/2575295303 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1c80024070 con 0x7f1c88107b00 2024-03-26T06:37:08.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.201+0000 7f1c857fa700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c740a2890 0x7f1c740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:08.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.201+0000 7f1c87fff700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c740a2890 0x7f1c740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:08.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.201+0000 7f1c857fa700 1 -- 172.21.15.38:0/2575295303 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f1c800cfbe0 con 0x7f1c88107b00 2024-03-26T06:37:08.205 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.202+0000 7f1c87fff700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c740a2890 0x7f1c740a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1c70005950 tx=0x7f1c700058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:08.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.205+0000 7f1c857fa700 1 -- 172.21.15.38:0/2575295303 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1c8009cd10 con 0x7f1c88107b00 2024-03-26T06:37:08.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.417+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/2575295303 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c88062220 con 0x7f1c740a2890 2024-03-26T06:37:08.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.419+0000 7f1c857fa700 1 -- 172.21.15.38:0/2575295303 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f1c88062220 con 0x7f1c740a2890 2024-03-26T06:37:08.422 INFO:teuthology.orchestra.run.smithi038.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-26T06:37:08.422 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN 3726G No 3m ago Has a FileSystem, LVM detected 2024-03-26T06:37:08.422 INFO:teuthology.orchestra.run.smithi038.stdout:smithi121 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400ZF400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-26T06:37:08.422 INFO:teuthology.orchestra.run.smithi038.stdout:smithi170 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002R480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-03-26T06:37:08.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/2575295303 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c740a2890 msgr2=0x7f1c740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:08.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c740a2890 0x7f1c740a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1c70005950 tx=0x7f1c700058e0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/2575295303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c88107b00 msgr2=0x7f1c88100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:08.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c88107b00 0x7f1c88100d90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000b070 tx=0x7f1c80004db0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/2575295303 shutdown_connections 2024-03-26T06:37:08.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c88107b00 0x7f1c88100d90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f1c881028a0 0x7f1c88102200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f1c740a2890 0x7f1c740a4d50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 --2- 172.21.15.38:0/2575295303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1c881051d0 0x7f1c88100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:08.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/2575295303 >> 172.21.15.38:0/2575295303 conn(0x7f1c880fc2b0 msgr2=0x7f1c881031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:08.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/2575295303 shutdown_connections 2024-03-26T06:37:08.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:08.421+0000 7f1c8e3c5700 1 -- 172.21.15.38:0/2575295303 wait complete. 2024-03-26T06:37:09.092 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T06:37:09.102 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T06:37:09.102 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-26T06:37:09.346 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:37:09.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:09 smithi170 ceph-mon[36912]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-26T06:37:09.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:09 smithi170 ceph-mon[36912]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:09.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:09 smithi038 ceph-mon[32265]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-26T06:37:09.678 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:09 smithi038 ceph-mon[32265]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:09.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:09 smithi121 ceph-mon[36349]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2024-03-26T06:37:09.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:09 smithi121 ceph-mon[36349]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:10.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.919+0000 7fe057924700 1 -- 172.21.15.38:0/4192987860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe050100d60 msgr2=0x7fe050101180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:10.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.919+0000 7fe057924700 1 --2- 172.21.15.38:0/4192987860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe050100d60 0x7fe050101180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe0400099c0 tx=0x7fe040009cd0 comp rx=0 tx=0).stop 2024-03-26T06:37:10.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.919+0000 7fe057924700 1 -- 172.21.15.38:0/4192987860 shutdown_connections 2024-03-26T06:37:10.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.919+0000 7fe057924700 1 --2- 172.21.15.38:0/4192987860 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050102900 0x7fe0501091b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:10.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.919+0000 7fe057924700 1 --2- 172.21.15.38:0/4192987860 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe050100d60 0x7fe050101180 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.919+0000 7fe057924700 1 --2- 172.21.15.38:0/4192987860 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe050101f60 0x7fe0501023c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.919+0000 7fe057924700 1 -- 172.21.15.38:0/4192987860 >> 172.21.15.38:0/4192987860 conn(0x7fe0500fc300 msgr2=0x7fe0500fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.919+0000 7fe057924700 1 -- 172.21.15.38:0/4192987860 shutdown_connections 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.919+0000 7fe057924700 1 -- 172.21.15.38:0/4192987860 wait complete. 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe057924700 1 Processor -- start 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe057924700 1 -- start start 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe057924700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050100d60 0x7fe050104ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe057924700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe050101f60 0x7fe050103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe057924700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe050102900 0x7fe050103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe057924700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05010c9a0 con 0x7fe050101f60 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe057924700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05010c820 con 0x7fe050100d60 2024-03-26T06:37:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe057924700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05010cb20 con 0x7fe050102900 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe054ebf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe050101f60 0x7fe050103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe054ebf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe050101f60 0x7fe050103340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43736/0 (socket says 172.21.15.38:43736) 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe054ebf700 1 -- 172.21.15.38:0/4064631613 learned_addr learned my addr 172.21.15.38:0/4064631613 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.920+0000 7fe0556c0700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050100d60 0x7fe050104ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe054ebf700 1 -- 172.21.15.38:0/4064631613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe050102900 msgr2=0x7fe050103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe055ec1700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe050102900 0x7fe050103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe054ebf700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe050102900 0x7fe050103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe054ebf700 1 -- 172.21.15.38:0/4064631613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050100d60 msgr2=0x7fe050104ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe054ebf700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050100d60 0x7fe050104ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe054ebf700 1 -- 172.21.15.38:0/4064631613 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe050104150 con 0x7fe050101f60 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe0556c0700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050100d60 0x7fe050104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:37:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe055ec1700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe050102900 0x7fe050103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe054ebf700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe050101f60 0x7fe050103340 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe04c00b530 tx=0x7fe04c00b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe0467fc700 1 -- 172.21.15.38:0/4064631613 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c015020 con 0x7fe050101f60 2024-03-26T06:37:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe0467fc700 1 -- 172.21.15.38:0/4064631613 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe04c011d10 con 0x7fe050101f60 2024-03-26T06:37:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe057924700 1 -- 172.21.15.38:0/4064631613 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0501a2210 con 0x7fe050101f60 2024-03-26T06:37:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe0467fc700 1 -- 172.21.15.38:0/4064631613 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c010e50 con 0x7fe050101f60 2024-03-26T06:37:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.921+0000 7fe057924700 1 -- 172.21.15.38:0/4064631613 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0501a2770 con 0x7fe050101f60 2024-03-26T06:37:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.922+0000 7fe057924700 1 -- 172.21.15.38:0/4064631613 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe050068aa0 con 0x7fe050101f60 2024-03-26T06:37:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.923+0000 7fe0467fc700 1 -- 172.21.15.38:0/4064631613 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe04c018020 con 0x7fe050101f60 2024-03-26T06:37:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.924+0000 7fe0467fc700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe03c0a2ab0 0x7fe03c0a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.924+0000 7fe0556c0700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe03c0a2ab0 0x7fe03c0a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.924+0000 7fe0467fc700 1 -- 172.21.15.38:0/4064631613 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe04c0ce8a0 con 0x7fe050101f60 2024-03-26T06:37:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.924+0000 7fe0556c0700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe03c0a2ab0 0x7fe03c0a4f70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe04000b580 tx=0x7fe040005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:10.930 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:10.927+0000 7fe0467fc700 1 -- 172.21.15.38:0/4064631613 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe04c01f020 con 0x7fe050101f60 2024-03-26T06:37:11.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.185+0000 7fe057924700 1 -- 172.21.15.38:0/4064631613 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe0501a2c00 con 0x7fe050101f60 2024-03-26T06:37:11.188 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.185+0000 7fe0467fc700 1 -- 172.21.15.38:0/4064631613 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 22 v22) v1 ==== 76+0+2943 (secure 0 0 0) 0x7fe04c09b900 con 0x7fe050101f60 2024-03-26T06:37:11.188 INFO:teuthology.orchestra.run.smithi038.stdout:e22 2024-03-26T06:37:11.188 INFO:teuthology.orchestra.run.smithi038.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T06:37:11.188 INFO:teuthology.orchestra.run.smithi038.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:legacy client fscid: 1 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:Filesystem 'cephfs' (1) 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:fs_name cephfs 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 22 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-03-26T06:35:26.477444+0000 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-03-26T06:35:32.167599+0000 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:tableserver 0 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:root 0 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:session_timeout 300 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:session_autoclose 300 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:max_file_size 1099511627776 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:max_xattr_size 65536 2024-03-26T06:37:11.189 INFO:teuthology.orchestra.run.smithi038.stdout:required_client_features {} 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure 0 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure_osd_epoch 0 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:max_mds 3 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:in 0,1,2 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:up {0=24539,1=24502,2=24488} 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:failed 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:damaged 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:stopped 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:data_pools [3] 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:metadata_pool 2 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:inline_data disabled 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:balancer 2024-03-26T06:37:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:bal_rank_mask -1 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout:standby_count_wanted 0 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.b{0:24539} state up:active seq 3 addr [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.c{0:24472} state up:standby-replay seq 1 addr [v2:172.21.15.170:6836/2484220351,v1:172.21.15.170:6837/2484220351] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.i{1:24502} state up:active seq 6 addr [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.e{1:24464} state up:standby-replay seq 1 addr [v2:172.21.15.121:6834/3384748966,v1:172.21.15.121:6835/3384748966] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.h{2:24488} state up:active seq 7 addr [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.f{2:24460} state up:standby-replay seq 1 addr [v2:172.21.15.170:6834/2374885191,v1:172.21.15.170:6835/2374885191] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout:Standby daemons: 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.j{-1:14556} state up:standby seq 1 addr [v2:172.21.15.38:6834/4105682236,v1:172.21.15.38:6835/4105682236] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.d{-1:14568} state up:standby seq 1 addr [v2:172.21.15.38:6836/3363372565,v1:172.21.15.38:6837/3363372565] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.191 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.g{-1:14628} state up:standby seq 1 addr [v2:172.21.15.38:6838/357411663,v1:172.21.15.38:6839/357411663] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.192 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.a{-1:14640} state up:standby seq 1 addr [v2:172.21.15.38:6840/654319512,v1:172.21.15.38:6841/654319512] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.192 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.l{-1:24412} state up:standby seq 1 addr [v2:172.21.15.170:6832/1336487813,v1:172.21.15.170:6833/1336487813] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.192 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.k{-1:24431} state up:standby seq 1 addr [v2:172.21.15.121:6832/2121798105,v1:172.21.15.121:6833/2121798105] compat {c=[1],r=[1],i=[fff]}] 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.188+0000 7fe057924700 1 -- 172.21.15.38:0/4064631613 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe03c0a2ab0 msgr2=0x7fe03c0a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.189+0000 7fe057924700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe03c0a2ab0 0x7fe03c0a4f70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fe04000b580 tx=0x7fe040005fb0 comp rx=0 tx=0).stop 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.189+0000 7fe057924700 1 -- 172.21.15.38:0/4064631613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe050101f60 msgr2=0x7fe050103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.189+0000 7fe057924700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe050101f60 0x7fe050103340 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe04c00b530 tx=0x7fe04c00b8f0 comp rx=0 tx=0).stop 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.189+0000 7fe057924700 1 -- 172.21.15.38:0/4064631613 shutdown_connections 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.189+0000 7fe057924700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe050102900 0x7fe050103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.189+0000 7fe057924700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe03c0a2ab0 0x7fe03c0a4f70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.189+0000 7fe057924700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe050100d60 0x7fe050104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.189+0000 7fe057924700 1 --2- 172.21.15.38:0/4064631613 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe050101f60 0x7fe050103340 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.189+0000 7fe057924700 1 -- 172.21.15.38:0/4064631613 >> 172.21.15.38:0/4064631613 conn(0x7fe0500fc300 msgr2=0x7fe0500fde00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.189+0000 7fe057924700 1 -- 172.21.15.38:0/4064631613 shutdown_connections 2024-03-26T06:37:11.193 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:11.189+0000 7fe057924700 1 -- 172.21.15.38:0/4064631613 wait complete. 2024-03-26T06:37:11.194 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 22 2024-03-26T06:37:11.514 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:11 smithi038 ceph-mon[32265]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-26T06:37:11.514 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:11 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/4064631613' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T06:37:11.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:11 smithi170 ceph-mon[36912]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-26T06:37:11.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:11 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/4064631613' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T06:37:11.697 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:11 smithi121 ceph-mon[36349]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-26T06:37:11.698 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:11 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/4064631613' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-26T06:37:11.817 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-26T06:37:12.061 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:37:13.492 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:13 smithi038 ceph-mon[32265]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-26T06:37:13.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:13 smithi170 ceph-mon[36912]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-26T06:37:13.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.729+0000 7fa6d20a6700 1 -- 172.21.15.38:0/1362486885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6cc102f70 msgr2=0x7fa6cc1033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:13.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.729+0000 7fa6d20a6700 1 --2- 172.21.15.38:0/1362486885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6cc102f70 0x7fa6cc1033f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009990 tx=0x7fa6c0009ca0 comp rx=0 tx=0).stop 2024-03-26T06:37:13.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.729+0000 7fa6d20a6700 1 -- 172.21.15.38:0/1362486885 shutdown_connections 2024-03-26T06:37:13.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.729+0000 7fa6d20a6700 1 --2- 172.21.15.38:0/1362486885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6cc103930 0x7fa6cc10a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:13.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.729+0000 7fa6d20a6700 1 --2- 172.21.15.38:0/1362486885 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6cc101e10 0x7fa6cc102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:13.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.729+0000 7fa6d20a6700 1 --2- 172.21.15.38:0/1362486885 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6cc102f70 0x7fa6cc1033f0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:13.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.729+0000 7fa6d20a6700 1 -- 172.21.15.38:0/1362486885 >> 172.21.15.38:0/1362486885 conn(0x7fa6cc0fd3f0 msgr2=0x7fa6cc0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:13.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.729+0000 7fa6d20a6700 1 -- 172.21.15.38:0/1362486885 shutdown_connections 2024-03-26T06:37:13.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.729+0000 7fa6d20a6700 1 -- 172.21.15.38:0/1362486885 wait complete. 2024-03-26T06:37:13.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.730+0000 7fa6d20a6700 1 Processor -- start 2024-03-26T06:37:13.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.730+0000 7fa6d20a6700 1 -- start start 2024-03-26T06:37:13.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.730+0000 7fa6d20a6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6cc101e10 0x7fa6cc198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:13.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.730+0000 7fa6d20a6700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6cc102f70 0x7fa6cc198690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:13.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.730+0000 7fa6d20a6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6cc103930 0x7fa6cc19f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:13.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.730+0000 7fa6d20a6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc10dae0 con 0x7fa6cc101e10 2024-03-26T06:37:13.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.730+0000 7fa6d20a6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc10d960 con 0x7fa6cc103930 2024-03-26T06:37:13.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.730+0000 7fa6d20a6700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6cc10dc60 con 0x7fa6cc102f70 2024-03-26T06:37:13.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.730+0000 7fa6cb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6cc101e10 0x7fa6cc198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:13.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.730+0000 7fa6cb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6cc101e10 0x7fa6cc198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43756/0 (socket says 172.21.15.38:43756) 2024-03-26T06:37:13.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.730+0000 7fa6cb7fe700 1 -- 172.21.15.38:0/463258489 learned_addr learned my addr 172.21.15.38:0/463258489 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:13.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6cbfff700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6cc103930 0x7fa6cc19f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6caffd700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6cc102f70 0x7fa6cc198690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6cb7fe700 1 -- 172.21.15.38:0/463258489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6cc102f70 msgr2=0x7fa6cc198690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6cb7fe700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6cc102f70 0x7fa6cc198690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6cb7fe700 1 -- 172.21.15.38:0/463258489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6cc103930 msgr2=0x7fa6cc19f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6cb7fe700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6cc103930 0x7fa6cc19f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6cb7fe700 1 -- 172.21.15.38:0/463258489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc19fdb0 con 0x7fa6cc101e10 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6caffd700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6cc102f70 0x7fa6cc198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6cb7fe700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6cc101e10 0x7fa6cc198150 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00b560 tx=0x7fa6bc00b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6c8ff9700 1 -- 172.21.15.38:0/463258489 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc015020 con 0x7fa6cc101e10 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6d20a6700 1 -- 172.21.15.38:0/463258489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6cc1a00a0 con 0x7fa6cc101e10 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6c8ff9700 1 -- 172.21.15.38:0/463258489 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6bc011d10 con 0x7fa6cc101e10 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6c8ff9700 1 -- 172.21.15.38:0/463258489 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc00f2e0 con 0x7fa6cc101e10 2024-03-26T06:37:13.736 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.731+0000 7fa6d20a6700 1 -- 172.21.15.38:0/463258489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6cc1a0660 con 0x7fa6cc101e10 2024-03-26T06:37:13.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.732+0000 7fa6d20a6700 1 -- 172.21.15.38:0/463258489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6cc068aa0 con 0x7fa6cc101e10 2024-03-26T06:37:13.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.733+0000 7fa6c8ff9700 1 -- 172.21.15.38:0/463258489 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa6bc018020 con 0x7fa6cc101e10 2024-03-26T06:37:13.737 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.734+0000 7fa6c8ff9700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa6b40a2890 0x7fa6b40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:13.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.734+0000 7fa6caffd700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa6b40a2890 0x7fa6b40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:13.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.734+0000 7fa6c8ff9700 1 -- 172.21.15.38:0/463258489 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fa6bc0ce250 con 0x7fa6cc101e10 2024-03-26T06:37:13.738 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.734+0000 7fa6caffd700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa6b40a2890 0x7fa6b40a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0000c00 tx=0x7fa6c0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:13.741 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.738+0000 7fa6c8ff9700 1 -- 172.21.15.38:0/463258489 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa6bc01f020 con 0x7fa6cc101e10 2024-03-26T06:37:13.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:13 smithi121 ceph-mon[36349]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-26T06:37:13.946 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.944+0000 7fa6d20a6700 1 -- 172.21.15.38:0/463258489 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fa6cc0629f0 con 0x7fa6cc101e10 2024-03-26T06:37:13.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.945+0000 7fa6c8ff9700 1 -- 172.21.15.38:0/463258489 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) v1 ==== 56+0+5308 (secure 0 0 0) 0x7fa6bc09b2f0 con 0x7fa6cc101e10 2024-03-26T06:37:13.947 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 88 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:fsid c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:created 2024-03-26T06:27:32.376242+0000 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2024-03-26T06:36:27.512655+0000 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:crush_version 26 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:full_ratio 0.95 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:backfillfull_ratio 0.9 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:nearfull_ratio 0.85 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:require_min_compat_client luminous 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:min_compat_client luminous 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:require_osd_release squid 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:stretch_mode_enabled false 2024-03-26T06:37:13.948 INFO:teuthology.orchestra.run.smithi038.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:max_osd 12 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6802/1449059243,v1:172.21.15.38:6803/1449059243] [v2:172.21.15.38:6804/1449059243,v1:172.21.15.38:6805/1449059243] exists,up 7ec203cd-dc26-4133-83b3-de1d661ddd0d 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6800/3506119522,v1:172.21.15.121:6801/3506119522] [v2:172.21.15.121:6802/3506119522,v1:172.21.15.121:6803/3506119522] exists,up 3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6800/1189530871,v1:172.21.15.170:6801/1189530871] [v2:172.21.15.170:6802/1189530871,v1:172.21.15.170:6803/1189530871] exists,up 09ebac77-3d71-4e0a-b07a-6797adda9dd0 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 up in weight 1 up_from 26 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6810/3343509291,v1:172.21.15.38:6811/3343509291] [v2:172.21.15.38:6812/3343509291,v1:172.21.15.38:6813/3343509291] exists,up 05893f6a-6566-412d-8341-72c9d577310b 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 up in weight 1 up_from 34 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6808/1471300176,v1:172.21.15.121:6809/1471300176] [v2:172.21.15.121:6810/1471300176,v1:172.21.15.121:6811/1471300176] exists,up 14ae6f68-9b15-4098-8bea-4c1688b4a0fb 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 up in weight 1 up_from 39 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6808/718927153,v1:172.21.15.170:6809/718927153] [v2:172.21.15.170:6810/718927153,v1:172.21.15.170:6811/718927153] exists,up 8bb8ecf7-e48a-4e5a-b313-f84660cf29f6 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6818/757778059,v1:172.21.15.38:6819/757778059] [v2:172.21.15.38:6820/757778059,v1:172.21.15.38:6821/757778059] exists,up 25f7a3a4-3f5f-4625-8f87-766305b481c4 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 up in weight 1 up_from 52 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6816/446835891,v1:172.21.15.121:6817/446835891] [v2:172.21.15.121:6818/446835891,v1:172.21.15.121:6819/446835891] exists,up 11318d82-7c42-4e41-81b9-ecf84b65cd61 2024-03-26T06:37:13.949 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8 up in weight 1 up_from 60 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6816/1764725971,v1:172.21.15.170:6817/1764725971] [v2:172.21.15.170:6818/1764725971,v1:172.21.15.170:6819/1764725971] exists,up fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6826/539511510,v1:172.21.15.38:6827/539511510] [v2:172.21.15.38:6828/539511510,v1:172.21.15.38:6829/539511510] exists,up 0fa2735f-8c3c-41bd-8de1-2681f659088c 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10 up in weight 1 up_from 73 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.121:6824/1310211853,v1:172.21.15.121:6825/1310211853] [v2:172.21.15.121:6826/1310211853,v1:172.21.15.121:6827/1310211853] exists,up fa25661a-1c05-40d1-a17c-94d274eb01f8 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.170:6824/3915107754,v1:172.21.15.170:6825/3915107754] [v2:172.21.15.170:6826/3915107754,v1:172.21.15.170:6827/3915107754] exists,up c870b245-cb9a-4275-9b4f-6f86c049991a 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.6 [8,11] 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.b [2,4] 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.14 [2,4] 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.1f [7,11] 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.11 [4,8] 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.20 [5,9] 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.2d [3,9] 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.31 [4,6] 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 3.36 [4,9] 2024-03-26T06:37:13.950 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/2527804067 expires 2024-03-27T06:28:26.137775+0000 2024-03-26T06:37:13.951 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6801/4246298230 expires 2024-03-27T06:28:26.137775+0000 2024-03-26T06:37:13.951 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/1523129198 expires 2024-03-27T06:28:26.137775+0000 2024-03-26T06:37:13.951 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/2579773966 expires 2024-03-27T06:27:59.817781+0000 2024-03-26T06:37:13.951 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6800/1985135432 expires 2024-03-27T06:27:59.817781+0000 2024-03-26T06:37:13.951 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/2512207794 expires 2024-03-27T06:27:59.817781+0000 2024-03-26T06:37:13.951 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6800/4246298230 expires 2024-03-27T06:28:26.137775+0000 2024-03-26T06:37:13.951 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/3929636377 expires 2024-03-27T06:27:59.817781+0000 2024-03-26T06:37:13.951 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6801/1985135432 expires 2024-03-27T06:27:59.817781+0000 2024-03-26T06:37:13.951 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/3896600378 expires 2024-03-27T06:28:26.137775+0000 2024-03-26T06:37:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 -- 172.21.15.38:0/463258489 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa6b40a2890 msgr2=0x7fa6b40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa6b40a2890 0x7fa6b40a4d50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0000c00 tx=0x7fa6c0005fb0 comp rx=0 tx=0).stop 2024-03-26T06:37:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 -- 172.21.15.38:0/463258489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6cc101e10 msgr2=0x7fa6cc198150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6cc101e10 0x7fa6cc198150 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00b560 tx=0x7fa6bc00b920 comp rx=0 tx=0).stop 2024-03-26T06:37:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 -- 172.21.15.38:0/463258489 shutdown_connections 2024-03-26T06:37:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fa6b40a2890 0x7fa6b40a4d50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6cc103930 0x7fa6cc19f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6cc101e10 0x7fa6cc198150 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 --2- 172.21.15.38:0/463258489 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa6cc102f70 0x7fa6cc198690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 -- 172.21.15.38:0/463258489 >> 172.21.15.38:0/463258489 conn(0x7fa6cc0fd3f0 msgr2=0x7fa6cc106380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:13.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 -- 172.21.15.38:0/463258489 shutdown_connections 2024-03-26T06:37:13.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:13.947+0000 7fa6d20a6700 1 -- 172.21.15.38:0/463258489 wait complete. 2024-03-26T06:37:14.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:14 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/463258489' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T06:37:14.675 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-26T06:37:14.687 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:37:14.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 -- 172.21.15.38:0/1135023163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ac102ec0 msgr2=0x7fe5ac109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:14.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 --2- 172.21.15.38:0/1135023163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ac102ec0 0x7fe5ac109750 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4009900 tx=0x7fe5a402bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:14.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 -- 172.21.15.38:0/1135023163 shutdown_connections 2024-03-26T06:37:14.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 --2- 172.21.15.38:0/1135023163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ac102ec0 0x7fe5ac109750 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:14.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 --2- 172.21.15.38:0/1135023163 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5ac102500 0x7fe5ac102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:14.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 --2- 172.21.15.38:0/1135023163 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac101310 0x7fe5ac101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:14.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 -- 172.21.15.38:0/1135023163 >> 172.21.15.38:0/1135023163 conn(0x7fe5ac0fc8a0 msgr2=0x7fe5ac0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:14.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 -- 172.21.15.38:0/1135023163 shutdown_connections 2024-03-26T06:37:14.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 -- 172.21.15.38:0/1135023163 wait complete. 2024-03-26T06:37:14.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 Processor -- start 2024-03-26T06:37:14.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 -- start start 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5ac101310 0x7fe5ac197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac102500 0x7fe5ac197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ac102ec0 0x7fe5ac19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac10d080 con 0x7fe5ac102ec0 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac10cf00 con 0x7fe5ac102500 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.792+0000 7fe5b1659640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ac10d200 con 0x7fe5ac101310 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5ab7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ac102ec0 0x7fe5ac19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5ab7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ac102ec0 0x7fe5ac19ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41982/0 (socket says 172.21.15.38:41982) 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5ab7fe640 1 -- 172.21.15.38:0/4245559747 learned_addr learned my addr 172.21.15.38:0/4245559747 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5aaffd640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5ac101310 0x7fe5ac197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5aa7fc640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac102500 0x7fe5ac197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:14.798 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5ab7fe640 1 -- 172.21.15.38:0/4245559747 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5ac101310 msgr2=0x7fe5ac197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5ab7fe640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5ac101310 0x7fe5ac197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5ab7fe640 1 -- 172.21.15.38:0/4245559747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac102500 msgr2=0x7fe5ac197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5ab7fe640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac102500 0x7fe5ac197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5ab7fe640 1 -- 172.21.15.38:0/4245559747 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ac19f2b0 con 0x7fe5ac102ec0 2024-03-26T06:37:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5ab7fe640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ac102ec0 0x7fe5ac19ec40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4005ab0 tx=0x7fe5a4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5a3fff640 1 -- 172.21.15.38:0/4245559747 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4041070 con 0x7fe5ac102ec0 2024-03-26T06:37:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5b1659640 1 -- 172.21.15.38:0/4245559747 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ac19f540 con 0x7fe5ac102ec0 2024-03-26T06:37:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5a3fff640 1 -- 172.21.15.38:0/4245559747 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5a40054e0 con 0x7fe5ac102ec0 2024-03-26T06:37:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5a3fff640 1 -- 172.21.15.38:0/4245559747 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a40346a0 con 0x7fe5ac102ec0 2024-03-26T06:37:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.793+0000 7fe5b1659640 1 -- 172.21.15.38:0/4245559747 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ac19fa80 con 0x7fe5ac102ec0 2024-03-26T06:37:14.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.794+0000 7fe5b1659640 1 -- 172.21.15.38:0/4245559747 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ac108630 con 0x7fe5ac102ec0 2024-03-26T06:37:14.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.795+0000 7fe5a3fff640 1 -- 172.21.15.38:0/4245559747 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe5a4044070 con 0x7fe5ac102ec0 2024-03-26T06:37:14.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.796+0000 7fe5a3fff640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe5980a2800 0x7fe5980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:14.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.796+0000 7fe5a3fff640 1 -- 172.21.15.38:0/4245559747 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe5a40f0f10 con 0x7fe5ac102ec0 2024-03-26T06:37:14.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.796+0000 7fe5aaffd640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe5980a2800 0x7fe5980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:14.800 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.796+0000 7fe5aaffd640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe5980a2800 0x7fe5980a4cc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac102360 tx=0x7fe59c00d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:14.802 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.800+0000 7fe5a3fff640 1 -- 172.21.15.38:0/4245559747 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe5a40bdf70 con 0x7fe5ac102ec0 2024-03-26T06:37:14.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:14 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/463258489' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T06:37:14.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:14 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/463258489' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T06:37:14.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.984+0000 7fe5b1659640 1 -- 172.21.15.38:0/4245559747 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5ac061ef0 con 0x7fe5ac102ec0 2024-03-26T06:37:14.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.986+0000 7fe5a3fff640 1 -- 172.21.15.38:0/4245559747 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22634 (secure 0 0 0) 0x7fe5a40bd870 con 0x7fe5ac102ec0 2024-03-26T06:37:14.988 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:37:14.989 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":88,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:36:27.512655+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:37:14.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 -- 172.21.15.38:0/4245559747 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe5980a2800 msgr2=0x7fe5980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:14.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe5980a2800 0x7fe5980a4cc0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe5ac102360 tx=0x7fe59c00d040 comp rx=0 tx=0).stop 2024-03-26T06:37:14.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 -- 172.21.15.38:0/4245559747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ac102ec0 msgr2=0x7fe5ac19ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:14.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ac102ec0 0x7fe5ac19ec40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4005ab0 tx=0x7fe5a4004270 comp rx=0 tx=0).stop 2024-03-26T06:37:14.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 -- 172.21.15.38:0/4245559747 shutdown_connections 2024-03-26T06:37:14.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe5980a2800 0x7fe5980a4cc0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:14.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5ac102ec0 0x7fe5ac19ec40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:14.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe5ac102500 0x7fe5ac197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:14.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 --2- 172.21.15.38:0/4245559747 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe5ac101310 0x7fe5ac197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:14.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 -- 172.21.15.38:0/4245559747 >> 172.21.15.38:0/4245559747 conn(0x7fe5ac0fc8a0 msgr2=0x7fe5ac105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:14.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 -- 172.21.15.38:0/4245559747 shutdown_connections 2024-03-26T06:37:14.991 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:14.988+0000 7fe5b1659640 1 -- 172.21.15.38:0/4245559747 wait complete. 2024-03-26T06:37:15.002 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T06:30:52.485897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-26T06:35:24.501259+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-26T06:35:25.675299+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T06:37:15.002 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T06:37:15.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.105+0000 7f6ea4b13640 1 -- 172.21.15.38:0/1335816729 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ea01013f0 msgr2=0x7f6ea0101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.105+0000 7f6ea4b13640 1 --2- 172.21.15.38:0/1335816729 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ea01013f0 0x7f6ea0101870 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009900 tx=0x7f6e8802bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:15.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.105+0000 7f6ea4b13640 1 -- 172.21.15.38:0/1335816729 shutdown_connections 2024-03-26T06:37:15.110 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.105+0000 7f6ea4b13640 1 --2- 172.21.15.38:0/1335816729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ea0101db0 0x7f6ea0108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.105+0000 7f6ea4b13640 1 --2- 172.21.15.38:0/1335816729 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ea01013f0 0x7f6ea0101870 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.105+0000 7f6ea4b13640 1 --2- 172.21.15.38:0/1335816729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ea0100200 0x7f6ea0100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.105+0000 7f6ea4b13640 1 -- 172.21.15.38:0/1335816729 >> 172.21.15.38:0/1335816729 conn(0x7f6ea00fb790 msgr2=0x7f6ea00fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.105+0000 7f6ea4b13640 1 -- 172.21.15.38:0/1335816729 shutdown_connections 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.105+0000 7f6ea4b13640 1 -- 172.21.15.38:0/1335816729 wait complete. 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6ea4b13640 1 Processor -- start 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6ea4b13640 1 -- start start 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6ea4b13640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ea0100200 0x7f6ea0197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6ea4b13640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ea01013f0 0x7f6ea0197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6ea4b13640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ea0101db0 0x7f6ea019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6ea4b13640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea010be90 con 0x7f6ea0101db0 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6ea4b13640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea010bd10 con 0x7f6ea01013f0 2024-03-26T06:37:15.111 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6ea4b13640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea010c010 con 0x7f6ea0100200 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6e9ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ea0101db0 0x7f6ea019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6e9ed9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ea0101db0 0x7f6ea019ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42000/0 (socket says 172.21.15.38:42000) 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6e9ed9d640 1 -- 172.21.15.38:0/2906634506 learned_addr learned my addr 172.21.15.38:0/2906634506 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6e97fff640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ea01013f0 0x7f6ea0197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6e9e59c640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ea0100200 0x7f6ea0197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6e9ed9d640 1 -- 172.21.15.38:0/2906634506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ea0100200 msgr2=0x7f6ea0197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6e9ed9d640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ea0100200 0x7f6ea0197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6e9ed9d640 1 -- 172.21.15.38:0/2906634506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ea01013f0 msgr2=0x7f6ea0197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6e9ed9d640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ea01013f0 0x7f6ea0197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6e9ed9d640 1 -- 172.21.15.38:0/2906634506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea019f240 con 0x7f6ea0101db0 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.106+0000 7f6e9ed9d640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ea0101db0 0x7f6ea019ebd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800cad0 tx=0x7f6e9800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.107+0000 7f6e977fe640 1 -- 172.21.15.38:0/2906634506 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9800ae20 con 0x7f6ea0101db0 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.107+0000 7f6ea4b13640 1 -- 172.21.15.38:0/2906634506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea019f530 con 0x7f6ea0101db0 2024-03-26T06:37:15.112 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.107+0000 7f6e977fe640 1 -- 172.21.15.38:0/2906634506 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e98005020 con 0x7f6ea0101db0 2024-03-26T06:37:15.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.107+0000 7f6e977fe640 1 -- 172.21.15.38:0/2906634506 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9800f6b0 con 0x7f6ea0101db0 2024-03-26T06:37:15.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.107+0000 7f6ea4b13640 1 -- 172.21.15.38:0/2906634506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea019faa0 con 0x7f6ea0101db0 2024-03-26T06:37:15.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.108+0000 7f6ea4b13640 1 -- 172.21.15.38:0/2906634506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea019fd60 con 0x7f6ea0101db0 2024-03-26T06:37:15.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.109+0000 7f6e977fe640 1 -- 172.21.15.38:0/2906634506 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6e980051d0 con 0x7f6ea0101db0 2024-03-26T06:37:15.113 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.109+0000 7f6e977fe640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6e8c0a2800 0x7f6e8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.109+0000 7f6e977fe640 1 -- 172.21.15.38:0/2906634506 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f6e980cede0 con 0x7f6ea0101db0 2024-03-26T06:37:15.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.109+0000 7f6e9e59c640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6e8c0a2800 0x7f6e8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.114 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.110+0000 7f6e9e59c640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6e8c0a2800 0x7f6e8c0a4cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0101250 tx=0x7f6e90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:15.116 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.114+0000 7f6e977fe640 1 -- 172.21.15.38:0/2906634506 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6e98014030 con 0x7f6ea0101db0 2024-03-26T06:37:15.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.296+0000 7f6ea4b13640 1 -- 172.21.15.38:0/2906634506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6ea019ff30 con 0x7f6ea0101db0 2024-03-26T06:37:15.299 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.297+0000 7f6e977fe640 1 -- 172.21.15.38:0/2906634506 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e9809be80 con 0x7f6ea0101db0 2024-03-26T06:37:15.299 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-03-26T06:37:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 -- 172.21.15.38:0/2906634506 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6e8c0a2800 msgr2=0x7f6e8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6e8c0a2800 0x7f6e8c0a4cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6ea0101250 tx=0x7f6e90005f90 comp rx=0 tx=0).stop 2024-03-26T06:37:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 -- 172.21.15.38:0/2906634506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ea0101db0 msgr2=0x7f6ea019ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ea0101db0 0x7f6ea019ebd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6e9800cad0 tx=0x7f6e9800cf90 comp rx=0 tx=0).stop 2024-03-26T06:37:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 -- 172.21.15.38:0/2906634506 shutdown_connections 2024-03-26T06:37:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f6e8c0a2800 0x7f6e8c0a4cc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ea0101db0 0x7f6ea019ebd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6ea01013f0 0x7f6ea0197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 --2- 172.21.15.38:0/2906634506 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6ea0100200 0x7f6ea0197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 -- 172.21.15.38:0/2906634506 >> 172.21.15.38:0/2906634506 conn(0x7f6ea00fb790 msgr2=0x7f6ea00fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 -- 172.21.15.38:0/2906634506 shutdown_connections 2024-03-26T06:37:15.303 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.299+0000 7f6ea4b13640 1 -- 172.21.15.38:0/2906634506 wait complete. 2024-03-26T06:37:15.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-26T06:37:15.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.416+0000 7fd95277e640 1 -- 172.21.15.38:0/3457137267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd94c1024f0 msgr2=0x7fd94c102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.416+0000 7fd95277e640 1 --2- 172.21.15.38:0/3457137267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd94c1024f0 0x7fd94c102970 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fd940009900 tx=0x7fd94002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:15.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.416+0000 7fd95277e640 1 -- 172.21.15.38:0/3457137267 shutdown_connections 2024-03-26T06:37:15.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.416+0000 7fd95277e640 1 --2- 172.21.15.38:0/3457137267 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd94c102eb0 0x7fd94c109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.416+0000 7fd95277e640 1 --2- 172.21.15.38:0/3457137267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd94c1024f0 0x7fd94c102970 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.416+0000 7fd95277e640 1 --2- 172.21.15.38:0/3457137267 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd94c101300 0x7fd94c101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.416+0000 7fd95277e640 1 -- 172.21.15.38:0/3457137267 >> 172.21.15.38:0/3457137267 conn(0x7fd94c0fc890 msgr2=0x7fd94c0fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:15.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.416+0000 7fd95277e640 1 -- 172.21.15.38:0/3457137267 shutdown_connections 2024-03-26T06:37:15.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.416+0000 7fd95277e640 1 -- 172.21.15.38:0/3457137267 wait complete. 2024-03-26T06:37:15.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.416+0000 7fd95277e640 1 Processor -- start 2024-03-26T06:37:15.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd95277e640 1 -- start start 2024-03-26T06:37:15.421 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd95277e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd94c101300 0x7fd94c197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd95277e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd94c1024f0 0x7fd94c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd95277e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd94c102eb0 0x7fd94c19eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd95277e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c10cf90 con 0x7fd94c1024f0 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd95277e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c10ce10 con 0x7fd94c102eb0 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd95277e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd94c10d110 con 0x7fd94c101300 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd94b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd94c1024f0 0x7fd94c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd94b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd94c1024f0 0x7fd94c197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42030/0 (socket says 172.21.15.38:42030) 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd94b7fe640 1 -- 172.21.15.38:0/3085349343 learned_addr learned my addr 172.21.15.38:0/3085349343 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd94bfff640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd94c101300 0x7fd94c197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd94b7fe640 1 -- 172.21.15.38:0/3085349343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd94c101300 msgr2=0x7fd94c197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd94b7fe640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd94c101300 0x7fd94c197590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.422 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd950d1b640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd94c102eb0 0x7fd94c19eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd94b7fe640 1 -- 172.21.15.38:0/3085349343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd94c102eb0 msgr2=0x7fd94c19eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd94b7fe640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd94c102eb0 0x7fd94c19eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd94b7fe640 1 -- 172.21.15.38:0/3085349343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd94c19f1b0 con 0x7fd94c1024f0 2024-03-26T06:37:15.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd94bfff640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd94c101300 0x7fd94c197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:15.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.417+0000 7fd94b7fe640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd94c1024f0 0x7fd94c197ad0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd940009230 tx=0x7fd940005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:15.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.418+0000 7fd9497fa640 1 -- 172.21.15.38:0/3085349343 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940041070 con 0x7fd94c1024f0 2024-03-26T06:37:15.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.418+0000 7fd9497fa640 1 -- 172.21.15.38:0/3085349343 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd9400054e0 con 0x7fd94c1024f0 2024-03-26T06:37:15.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.418+0000 7fd95277e640 1 -- 172.21.15.38:0/3085349343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd94c19f440 con 0x7fd94c1024f0 2024-03-26T06:37:15.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.418+0000 7fd9497fa640 1 -- 172.21.15.38:0/3085349343 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940034440 con 0x7fd94c1024f0 2024-03-26T06:37:15.423 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.418+0000 7fd95277e640 1 -- 172.21.15.38:0/3085349343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd94c19f930 con 0x7fd94c1024f0 2024-03-26T06:37:15.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.419+0000 7fd95277e640 1 -- 172.21.15.38:0/3085349343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd94c108620 con 0x7fd94c1024f0 2024-03-26T06:37:15.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.420+0000 7fd9497fa640 1 -- 172.21.15.38:0/3085349343 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd940044070 con 0x7fd94c1024f0 2024-03-26T06:37:15.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.420+0000 7fd9497fa640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd9340a2800 0x7fd9340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.424 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.420+0000 7fd94bfff640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd9340a2800 0x7fd9340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.420+0000 7fd9497fa640 1 -- 172.21.15.38:0/3085349343 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd9400f0bf0 con 0x7fd94c1024f0 2024-03-26T06:37:15.425 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.421+0000 7fd94bfff640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd9340a2800 0x7fd9340a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd94c102350 tx=0x7fd93c005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:15.426 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.424+0000 7fd9497fa640 1 -- 172.21.15.38:0/3085349343 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd9400bdc90 con 0x7fd94c1024f0 2024-03-26T06:37:15.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:15 smithi038 ceph-mon[32265]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:37:15.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:15 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/4245559747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:37:15.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:15 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2906634506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:37:15.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.607+0000 7fd95277e640 1 -- 172.21.15.38:0/3085349343 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd94c02d000 con 0x7fd94c1024f0 2024-03-26T06:37:15.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.607+0000 7fd9497fa640 1 -- 172.21.15.38:0/3085349343 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd9400bd590 con 0x7fd94c1024f0 2024-03-26T06:37:15.610 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-03-26T06:37:15.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 -- 172.21.15.38:0/3085349343 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd9340a2800 msgr2=0x7fd9340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd9340a2800 0x7fd9340a4cc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd94c102350 tx=0x7fd93c005db0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 -- 172.21.15.38:0/3085349343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd94c1024f0 msgr2=0x7fd94c197ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd94c1024f0 0x7fd94c197ad0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fd940009230 tx=0x7fd940005dd0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 -- 172.21.15.38:0/3085349343 shutdown_connections 2024-03-26T06:37:15.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd9340a2800 0x7fd9340a4cc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.612 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd94c102eb0 0x7fd94c19eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd94c1024f0 0x7fd94c197ad0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 --2- 172.21.15.38:0/3085349343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd94c101300 0x7fd94c197590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 -- 172.21.15.38:0/3085349343 >> 172.21.15.38:0/3085349343 conn(0x7fd94c0fc890 msgr2=0x7fd94c1057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:15.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 -- 172.21.15.38:0/3085349343 shutdown_connections 2024-03-26T06:37:15.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.609+0000 7fd95277e640 1 -- 172.21.15.38:0/3085349343 wait complete. 2024-03-26T06:37:15.622 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-26T06:37:15.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.725+0000 7fc36f545640 1 -- 172.21.15.38:0/4169359760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc368103530 msgr2=0x7fc368103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.725+0000 7fc36f545640 1 --2- 172.21.15.38:0/4169359760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc368103530 0x7fc368103950 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc360009900 tx=0x7fc36002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:15.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.725+0000 7fc36f545640 1 -- 172.21.15.38:0/4169359760 shutdown_connections 2024-03-26T06:37:15.730 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.725+0000 7fc36f545640 1 --2- 172.21.15.38:0/4169359760 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3681050e0 0x7fc36810b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.725+0000 7fc36f545640 1 --2- 172.21.15.38:0/4169359760 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc368104720 0x7fc368104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.725+0000 7fc36f545640 1 --2- 172.21.15.38:0/4169359760 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc368103530 0x7fc368103950 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.725+0000 7fc36f545640 1 -- 172.21.15.38:0/4169359760 >> 172.21.15.38:0/4169359760 conn(0x7fc3680feaa0 msgr2=0x7fc368100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.725+0000 7fc36f545640 1 -- 172.21.15.38:0/4169359760 shutdown_connections 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36f545640 1 -- 172.21.15.38:0/4169359760 wait complete. 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36f545640 1 Processor -- start 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36f545640 1 -- start start 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36f545640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc368103530 0x7fc368199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36f545640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc368104720 0x7fc368199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36f545640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3681050e0 0x7fc3681a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36f545640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36810f1c0 con 0x7fc3681050e0 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36f545640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36810f040 con 0x7fc368104720 2024-03-26T06:37:15.731 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36f545640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36810f340 con 0x7fc368103530 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36dae2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3681050e0 0x7fc3681a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36dae2640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3681050e0 0x7fc3681a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42056/0 (socket says 172.21.15.38:42056) 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36dae2640 1 -- 172.21.15.38:0/2284176867 learned_addr learned my addr 172.21.15.38:0/2284176867 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.726+0000 7fc36d2e1640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc368103530 0x7fc368199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc36cae0640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc368104720 0x7fc368199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc36dae2640 1 -- 172.21.15.38:0/2284176867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc368103530 msgr2=0x7fc368199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc36dae2640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc368103530 0x7fc368199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc36dae2640 1 -- 172.21.15.38:0/2284176867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc368104720 msgr2=0x7fc368199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc36dae2640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc368104720 0x7fc368199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc36dae2640 1 -- 172.21.15.38:0/2284176867 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3681a1490 con 0x7fc3681050e0 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc36dae2640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3681050e0 0x7fc3681a0e20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc35800cad0 tx=0x7fc35800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc3667fc640 1 -- 172.21.15.38:0/2284176867 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35800ae20 con 0x7fc3681050e0 2024-03-26T06:37:15.732 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc3667fc640 1 -- 172.21.15.38:0/2284176867 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc358005020 con 0x7fc3681050e0 2024-03-26T06:37:15.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc36f545640 1 -- 172.21.15.38:0/2284176867 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3681a1780 con 0x7fc3681050e0 2024-03-26T06:37:15.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc3667fc640 1 -- 172.21.15.38:0/2284176867 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35800f6f0 con 0x7fc3681050e0 2024-03-26T06:37:15.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.727+0000 7fc36f545640 1 -- 172.21.15.38:0/2284176867 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3681a1d20 con 0x7fc3681050e0 2024-03-26T06:37:15.733 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.728+0000 7fc36f545640 1 -- 172.21.15.38:0/2284176867 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36810a870 con 0x7fc3681050e0 2024-03-26T06:37:15.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.729+0000 7fc3667fc640 1 -- 172.21.15.38:0/2284176867 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc3580051d0 con 0x7fc3681050e0 2024-03-26T06:37:15.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.730+0000 7fc3667fc640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc3500a2880 0x7fc3500a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:15.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.730+0000 7fc36d2e1640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc3500a2880 0x7fc3500a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:15.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.730+0000 7fc3667fc640 1 -- 172.21.15.38:0/2284176867 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fc3580ced90 con 0x7fc3681050e0 2024-03-26T06:37:15.734 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.730+0000 7fc36d2e1640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc3500a2880 0x7fc3500a4d40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc360000c00 tx=0x7fc360005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:15.735 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.733+0000 7fc3667fc640 1 -- 172.21.15.38:0/2284176867 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc358014030 con 0x7fc3681050e0 2024-03-26T06:37:15.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:15 smithi121 ceph-mon[36349]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:37:15.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:15 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/4245559747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:37:15.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:15 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2906634506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:37:15.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:15 smithi170 ceph-mon[36912]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:37:15.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:15 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/4245559747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:37:15.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:15 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2906634506' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:37:15.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.917+0000 7fc36f545640 1 -- 172.21.15.38:0/2284176867 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc36802ce70 con 0x7fc3681050e0 2024-03-26T06:37:15.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.917+0000 7fc3667fc640 1 -- 172.21.15.38:0/2284176867 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc35809be70 con 0x7fc3681050e0 2024-03-26T06:37:15.920 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-03-26T06:37:15.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.919+0000 7fc36f545640 1 -- 172.21.15.38:0/2284176867 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc3500a2880 msgr2=0x7fc3500a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.919+0000 7fc36f545640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc3500a2880 0x7fc3500a4d40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc360000c00 tx=0x7fc360005fd0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.919+0000 7fc36f545640 1 -- 172.21.15.38:0/2284176867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3681050e0 msgr2=0x7fc3681a0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.919+0000 7fc36f545640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3681050e0 0x7fc3681a0e20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fc35800cad0 tx=0x7fc35800cf90 comp rx=0 tx=0).stop 2024-03-26T06:37:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.919+0000 7fc36f545640 1 -- 172.21.15.38:0/2284176867 shutdown_connections 2024-03-26T06:37:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.919+0000 7fc36f545640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fc3500a2880 0x7fc3500a4d40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.919+0000 7fc36f545640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3681050e0 0x7fc3681a0e20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.919+0000 7fc36f545640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc368104720 0x7fc368199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.919+0000 7fc36f545640 1 --2- 172.21.15.38:0/2284176867 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc368103530 0x7fc368199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.919+0000 7fc36f545640 1 -- 172.21.15.38:0/2284176867 >> 172.21.15.38:0/2284176867 conn(0x7fc3680feaa0 msgr2=0x7fc368107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.920+0000 7fc36f545640 1 -- 172.21.15.38:0/2284176867 shutdown_connections 2024-03-26T06:37:15.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:15.920+0000 7fc36f545640 1 -- 172.21.15.38:0/2284176867 wait complete. 2024-03-26T06:37:15.933 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:37:16.037 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.033+0000 7ffa72929640 1 -- 172.21.15.38:0/1686394517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa6c101e10 msgr2=0x7ffa6c104200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.033+0000 7ffa72929640 1 --2- 172.21.15.38:0/1686394517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa6c101e10 0x7ffa6c104200 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ffa60009900 tx=0x7ffa6002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:16.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.033+0000 7ffa72929640 1 -- 172.21.15.38:0/1686394517 shutdown_connections 2024-03-26T06:37:16.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.033+0000 7ffa72929640 1 --2- 172.21.15.38:0/1686394517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa6c107070 0x7ffa6c1094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.033+0000 7ffa72929640 1 --2- 172.21.15.38:0/1686394517 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa6c104740 0x7ffa6c106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.033+0000 7ffa72929640 1 --2- 172.21.15.38:0/1686394517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa6c101e10 0x7ffa6c104200 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.033+0000 7ffa72929640 1 -- 172.21.15.38:0/1686394517 >> 172.21.15.38:0/1686394517 conn(0x7ffa6c0fb820 msgr2=0x7ffa6c0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:16.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.033+0000 7ffa72929640 1 -- 172.21.15.38:0/1686394517 shutdown_connections 2024-03-26T06:37:16.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.033+0000 7ffa72929640 1 -- 172.21.15.38:0/1686394517 wait complete. 2024-03-26T06:37:16.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa72929640 1 Processor -- start 2024-03-26T06:37:16.038 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa72929640 1 -- start start 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa72929640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa6c101e10 0x7ffa6c101780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa72929640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa6c104740 0x7ffa6c0ffdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa72929640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa6c107070 0x7ffa6c100300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa72929640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c10cfb0 con 0x7ffa6c104740 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa72929640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c10ce30 con 0x7ffa6c107070 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa72929640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c10d130 con 0x7ffa6c101e10 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa6b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa6c104740 0x7ffa6c0ffdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa6b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa6c104740 0x7ffa6c0ffdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42082/0 (socket says 172.21.15.38:42082) 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa6b7fe640 1 -- 172.21.15.38:0/3409550829 learned_addr learned my addr 172.21.15.38:0/3409550829 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa6bfff640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa6c101e10 0x7ffa6c101780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa6b7fe640 1 -- 172.21.15.38:0/3409550829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa6c101e10 msgr2=0x7ffa6c101780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa6b7fe640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa6c101e10 0x7ffa6c101780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa70ec6640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa6c107070 0x7ffa6c100300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.039 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa6b7fe640 1 -- 172.21.15.38:0/3409550829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa6c107070 msgr2=0x7ffa6c100300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa6b7fe640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa6c107070 0x7ffa6c100300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.034+0000 7ffa6b7fe640 1 -- 172.21.15.38:0/3409550829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c100b90 con 0x7ffa6c104740 2024-03-26T06:37:16.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.035+0000 7ffa6b7fe640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa6c104740 0x7ffa6c0ffdc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00c970 tx=0x7ffa5c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:16.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.035+0000 7ffa697fa640 1 -- 172.21.15.38:0/3409550829 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c014070 con 0x7ffa6c104740 2024-03-26T06:37:16.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.035+0000 7ffa697fa640 1 -- 172.21.15.38:0/3409550829 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa5c004440 con 0x7ffa6c104740 2024-03-26T06:37:16.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.035+0000 7ffa72929640 1 -- 172.21.15.38:0/3409550829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6c1a16d0 con 0x7ffa6c104740 2024-03-26T06:37:16.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.035+0000 7ffa697fa640 1 -- 172.21.15.38:0/3409550829 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c00f450 con 0x7ffa6c104740 2024-03-26T06:37:16.040 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.035+0000 7ffa72929640 1 -- 172.21.15.38:0/3409550829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6c1a1c20 con 0x7ffa6c104740 2024-03-26T06:37:16.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.036+0000 7ffa72929640 1 -- 172.21.15.38:0/3409550829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6c0fd310 con 0x7ffa6c104740 2024-03-26T06:37:16.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.037+0000 7ffa697fa640 1 -- 172.21.15.38:0/3409550829 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffa5c017070 con 0x7ffa6c104740 2024-03-26T06:37:16.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.038+0000 7ffa697fa640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ffa540a2800 0x7ffa540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.038+0000 7ffa6bfff640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ffa540a2800 0x7ffa540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.038+0000 7ffa697fa640 1 -- 172.21.15.38:0/3409550829 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ffa5c0ce9c0 con 0x7ffa6c104740 2024-03-26T06:37:16.041 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.038+0000 7ffa6bfff640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ffa540a2800 0x7ffa540a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ffa60006010 tx=0x7ffa60000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:16.045 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.042+0000 7ffa697fa640 1 -- 172.21.15.38:0/3409550829 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ffa5c01e020 con 0x7ffa6c104740 2024-03-26T06:37:16.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.264+0000 7ffa72929640 1 -- 172.21.15.38:0/3409550829 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa6c02ce70 con 0x7ffa6c104740 2024-03-26T06:37:16.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.265+0000 7ffa697fa640 1 -- 172.21.15.38:0/3409550829 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11346 (secure 0 0 0) 0x7ffa5c09ba60 con 0x7ffa6c104740 2024-03-26T06:37:16.267 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:37:16.268 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/357411663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":357411663},{"type":"v1","addr":"172.21.15.38:6839","nonce":357411663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/654319512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":654319512},{"type":"v1","addr":"172.21.15.38:6841","nonce":654319512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-26T06:35:26.477444+0000","modified":"2024-03-26T06:35:32.167599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24502,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/359437185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":359437185},{"type":"v1","addr":"172.21.15.121:6839","nonce":359437185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-26T06:37:16.268 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 22 2024-03-26T06:37:16.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 -- 172.21.15.38:0/3409550829 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ffa540a2800 msgr2=0x7ffa540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ffa540a2800 0x7ffa540a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ffa60006010 tx=0x7ffa60000bc0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 -- 172.21.15.38:0/3409550829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa6c104740 msgr2=0x7ffa6c0ffdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa6c104740 0x7ffa6c0ffdc0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00c970 tx=0x7ffa5c00ce30 comp rx=0 tx=0).stop 2024-03-26T06:37:16.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 -- 172.21.15.38:0/3409550829 shutdown_connections 2024-03-26T06:37:16.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ffa540a2800 0x7ffa540a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffa6c107070 0x7ffa6c100300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffa6c104740 0x7ffa6c0ffdc0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 --2- 172.21.15.38:0/3409550829 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ffa6c101e10 0x7ffa6c101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 -- 172.21.15.38:0/3409550829 >> 172.21.15.38:0/3409550829 conn(0x7ffa6c0fb820 msgr2=0x7ffa6c102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:16.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 -- 172.21.15.38:0/3409550829 shutdown_connections 2024-03-26T06:37:16.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.267+0000 7ffa72929640 1 -- 172.21.15.38:0/3409550829 wait complete. 2024-03-26T06:37:16.282 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:37:16.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.382+0000 7feaa9f9d640 1 -- 172.21.15.38:0/1467458600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feaa4107070 msgr2=0x7feaa41094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.382+0000 7feaa9f9d640 1 --2- 172.21.15.38:0/1467458600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feaa4107070 0x7feaa41094c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fea9c009900 tx=0x7fea9c02bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:16.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.382+0000 7feaa9f9d640 1 -- 172.21.15.38:0/1467458600 shutdown_connections 2024-03-26T06:37:16.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.382+0000 7feaa9f9d640 1 --2- 172.21.15.38:0/1467458600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feaa4107070 0x7feaa41094c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.382+0000 7feaa9f9d640 1 --2- 172.21.15.38:0/1467458600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaa4104740 0x7feaa4106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.382+0000 7feaa9f9d640 1 --2- 172.21.15.38:0/1467458600 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feaa4101e10 0x7feaa4104200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.382+0000 7feaa9f9d640 1 -- 172.21.15.38:0/1467458600 >> 172.21.15.38:0/1467458600 conn(0x7feaa40fb820 msgr2=0x7feaa40fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:16.387 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.382+0000 7feaa9f9d640 1 -- 172.21.15.38:0/1467458600 shutdown_connections 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.382+0000 7feaa9f9d640 1 -- 172.21.15.38:0/1467458600 wait complete. 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.382+0000 7feaa9f9d640 1 Processor -- start 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.382+0000 7feaa9f9d640 1 -- start start 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa9f9d640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feaa4101e10 0x7feaa4101780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa9f9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaa4104740 0x7feaa40ffdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa9f9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feaa4107070 0x7feaa4100300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa9f9d640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa410d090 con 0x7feaa4107070 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa9f9d640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa410cf10 con 0x7feaa4104740 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa9f9d640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa410d210 con 0x7feaa4101e10 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feaa4107070 0x7feaa4100300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa3fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feaa4107070 0x7feaa4100300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42088/0 (socket says 172.21.15.38:42088) 2024-03-26T06:37:16.388 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa3fff640 1 -- 172.21.15.38:0/3612921934 learned_addr learned my addr 172.21.15.38:0/3612921934 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa2ffd640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaa4104740 0x7feaa40ffdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa37fe640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feaa4101e10 0x7feaa4101780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa3fff640 1 -- 172.21.15.38:0/3612921934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feaa4101e10 msgr2=0x7feaa4101780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa3fff640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feaa4101e10 0x7feaa4101780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa3fff640 1 -- 172.21.15.38:0/3612921934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaa4104740 msgr2=0x7feaa40ffdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa3fff640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaa4104740 0x7feaa40ffdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa3fff640 1 -- 172.21.15.38:0/3612921934 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa4100b90 con 0x7feaa4107070 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa3fff640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feaa4107070 0x7feaa4100300 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fea9c005ab0 tx=0x7fea9c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa0ff9640 1 -- 172.21.15.38:0/3612921934 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c041070 con 0x7feaa4107070 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa0ff9640 1 -- 172.21.15.38:0/3612921934 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea9c0054e0 con 0x7feaa4107070 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa9f9d640 1 -- 172.21.15.38:0/3612921934 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa41a1740 con 0x7feaa4107070 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa0ff9640 1 -- 172.21.15.38:0/3612921934 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c0346a0 con 0x7feaa4107070 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.383+0000 7feaa9f9d640 1 -- 172.21.15.38:0/3612921934 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa41a1c90 con 0x7feaa4107070 2024-03-26T06:37:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.384+0000 7feaa9f9d640 1 -- 172.21.15.38:0/3612921934 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa40fd310 con 0x7feaa4107070 2024-03-26T06:37:16.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.385+0000 7feaa0ff9640 1 -- 172.21.15.38:0/3612921934 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fea9c044070 con 0x7feaa4107070 2024-03-26T06:37:16.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.386+0000 7feaa0ff9640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fea8c0a2800 0x7fea8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.386+0000 7feaa37fe640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fea8c0a2800 0x7fea8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.386+0000 7feaa0ff9640 1 -- 172.21.15.38:0/3612921934 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fea9c0f0f80 con 0x7feaa4107070 2024-03-26T06:37:16.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.387+0000 7feaa37fe640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fea8c0a2800 0x7fea8c0a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fea94007dc0 tx=0x7fea940072f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:16.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.390+0000 7feaa0ff9640 1 -- 172.21.15.38:0/3612921934 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fea9c0be020 con 0x7feaa4107070 2024-03-26T06:37:16.578 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.575+0000 7feaa9f9d640 1 -- 172.21.15.38:0/3612921934 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feaa402d040 con 0x7feaa4107070 2024-03-26T06:37:16.579 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.577+0000 7feaa0ff9640 1 -- 172.21.15.38:0/3612921934 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22634 (secure 0 0 0) 0x7fea9c0bd920 con 0x7feaa4107070 2024-03-26T06:37:16.580 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:37:16.580 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":88,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:36:27.512655+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:37:16.581 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 -- 172.21.15.38:0/3612921934 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fea8c0a2800 msgr2=0x7fea8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fea8c0a2800 0x7fea8c0a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fea94007dc0 tx=0x7fea940072f0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 -- 172.21.15.38:0/3612921934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feaa4107070 msgr2=0x7feaa4100300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feaa4107070 0x7feaa4100300 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fea9c005ab0 tx=0x7fea9c004270 comp rx=0 tx=0).stop 2024-03-26T06:37:16.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 -- 172.21.15.38:0/3612921934 shutdown_connections 2024-03-26T06:37:16.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fea8c0a2800 0x7fea8c0a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feaa4107070 0x7feaa4100300 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feaa4104740 0x7feaa40ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 --2- 172.21.15.38:0/3612921934 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feaa4101e10 0x7feaa4101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 -- 172.21.15.38:0/3612921934 >> 172.21.15.38:0/3612921934 conn(0x7feaa40fb820 msgr2=0x7feaa4102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:16.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 -- 172.21.15.38:0/3612921934 shutdown_connections 2024-03-26T06:37:16.582 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.579+0000 7feaa9f9d640 1 -- 172.21.15.38:0/3612921934 wait complete. 2024-03-26T06:37:16.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:16 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3085349343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:37:16.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:16 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2284176867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:37:16.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:16 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3409550829' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:37:16.593 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T06:30:52.485897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-26T06:35:24.501259+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-26T06:35:25.675299+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T06:37:16.594 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T06:37:16.701 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.697+0000 7fbe91b48640 1 -- 172.21.15.38:0/2567950558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe8c1024f0 msgr2=0x7fbe8c102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.697+0000 7fbe91b48640 1 --2- 172.21.15.38:0/2567950558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe8c1024f0 0x7fbe8c102970 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbe80009900 tx=0x7fbe8002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.697+0000 7fbe91b48640 1 -- 172.21.15.38:0/2567950558 shutdown_connections 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.697+0000 7fbe91b48640 1 --2- 172.21.15.38:0/2567950558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe8c102eb0 0x7fbe8c109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.697+0000 7fbe91b48640 1 --2- 172.21.15.38:0/2567950558 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe8c1024f0 0x7fbe8c102970 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.697+0000 7fbe91b48640 1 --2- 172.21.15.38:0/2567950558 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe8c101300 0x7fbe8c101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.697+0000 7fbe91b48640 1 -- 172.21.15.38:0/2567950558 >> 172.21.15.38:0/2567950558 conn(0x7fbe8c0fc890 msgr2=0x7fbe8c0fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.697+0000 7fbe91b48640 1 -- 172.21.15.38:0/2567950558 shutdown_connections 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.697+0000 7fbe91b48640 1 -- 172.21.15.38:0/2567950558 wait complete. 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.697+0000 7fbe91b48640 1 Processor -- start 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.697+0000 7fbe91b48640 1 -- start start 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe91b48640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe8c101300 0x7fbe8c197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.702 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe91b48640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe8c1024f0 0x7fbe8c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe91b48640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe8c102eb0 0x7fbe8c19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe91b48640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8c10cfe0 con 0x7fbe8c101300 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe91b48640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8c10ce60 con 0x7fbe8c1024f0 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe91b48640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8c10d160 con 0x7fbe8c102eb0 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe8c101300 0x7fbe8c197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8b7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe8c101300 0x7fbe8c197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42110/0 (socket says 172.21.15.38:42110) 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8b7fe640 1 -- 172.21.15.38:0/3107079890 learned_addr learned my addr 172.21.15.38:0/3107079890 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8bfff640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe8c102eb0 0x7fbe8c19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8affd640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe8c1024f0 0x7fbe8c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8b7fe640 1 -- 172.21.15.38:0/3107079890 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe8c102eb0 msgr2=0x7fbe8c19ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8b7fe640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe8c102eb0 0x7fbe8c19ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8b7fe640 1 -- 172.21.15.38:0/3107079890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe8c1024f0 msgr2=0x7fbe8c197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8b7fe640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe8c1024f0 0x7fbe8c197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.703 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8b7fe640 1 -- 172.21.15.38:0/3107079890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8c19f250 con 0x7fbe8c101300 2024-03-26T06:37:16.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8bfff640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe8c102eb0 0x7fbe8c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:37:16.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8affd640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe8c1024f0 0x7fbe8c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:16.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe8b7fe640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe8c101300 0x7fbe8c197630 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c00e9c0 tx=0x7fbe7c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:16.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe88ff9640 1 -- 172.21.15.38:0/3107079890 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c00cd80 con 0x7fbe8c101300 2024-03-26T06:37:16.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe88ff9640 1 -- 172.21.15.38:0/3107079890 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe7c0035f0 con 0x7fbe8c101300 2024-03-26T06:37:16.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe91b48640 1 -- 172.21.15.38:0/3107079890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe8c19f540 con 0x7fbe8c101300 2024-03-26T06:37:16.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe88ff9640 1 -- 172.21.15.38:0/3107079890 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c010690 con 0x7fbe8c101300 2024-03-26T06:37:16.704 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.698+0000 7fbe91b48640 1 -- 172.21.15.38:0/3107079890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe8c19fa40 con 0x7fbe8c101300 2024-03-26T06:37:16.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.699+0000 7fbe91b48640 1 -- 172.21.15.38:0/3107079890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe8c108620 con 0x7fbe8c101300 2024-03-26T06:37:16.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.700+0000 7fbe88ff9640 1 -- 172.21.15.38:0/3107079890 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbe7c0037a0 con 0x7fbe8c101300 2024-03-26T06:37:16.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.701+0000 7fbe88ff9640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbe740a2800 0x7fbe740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:16.705 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.701+0000 7fbe8affd640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbe740a2800 0x7fbe740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:16.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.701+0000 7fbe88ff9640 1 -- 172.21.15.38:0/3107079890 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fbe7c0cef10 con 0x7fbe8c101300 2024-03-26T06:37:16.706 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.702+0000 7fbe8affd640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbe740a2800 0x7fbe740a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbe800096c0 tx=0x7fbe8002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:16.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.706+0000 7fbe88ff9640 1 -- 172.21.15.38:0/3107079890 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbe7c09bfb0 con 0x7fbe8c101300 2024-03-26T06:37:16.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.888+0000 7fbe91b48640 1 -- 172.21.15.38:0/3107079890 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe8c061ef0 con 0x7fbe8c101300 2024-03-26T06:37:16.891 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.888+0000 7fbe88ff9640 1 -- 172.21.15.38:0/3107079890 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbe7c09b8b0 con 0x7fbe8c101300 2024-03-26T06:37:16.892 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-03-26T06:37:16.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 -- 172.21.15.38:0/3107079890 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbe740a2800 msgr2=0x7fbe740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.893 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbe740a2800 0x7fbe740a4cc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbe800096c0 tx=0x7fbe8002df90 comp rx=0 tx=0).stop 2024-03-26T06:37:16.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 -- 172.21.15.38:0/3107079890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe8c101300 msgr2=0x7fbe8c197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:16.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe8c101300 0x7fbe8c197630 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c00e9c0 tx=0x7fbe7c00ee80 comp rx=0 tx=0).stop 2024-03-26T06:37:16.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 -- 172.21.15.38:0/3107079890 shutdown_connections 2024-03-26T06:37:16.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fbe740a2800 0x7fbe740a4cc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fbe8c102eb0 0x7fbe8c19ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe8c1024f0 0x7fbe8c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 --2- 172.21.15.38:0/3107079890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbe8c101300 0x7fbe8c197630 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:16.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 -- 172.21.15.38:0/3107079890 >> 172.21.15.38:0/3107079890 conn(0x7fbe8c0fc890 msgr2=0x7fbe8c1057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:16.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 -- 172.21.15.38:0/3107079890 shutdown_connections 2024-03-26T06:37:16.894 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:16.891+0000 7fbe91b48640 1 -- 172.21.15.38:0/3107079890 wait complete. 2024-03-26T06:37:16.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:16 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3085349343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:37:16.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:16 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2284176867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:37:16.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:16 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3409550829' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:37:16.905 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-26T06:37:16.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:16 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3085349343' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:37:16.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:16 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2284176867' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:37:16.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:16 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3409550829' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:37:17.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.008+0000 7fb7449d4640 1 -- 172.21.15.38:0/1481435460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb740101300 msgr2=0x7fb740101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.008+0000 7fb7449d4640 1 --2- 172.21.15.38:0/1481435460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb740101300 0x7fb740101720 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fb730009900 tx=0x7fb73002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:17.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.008+0000 7fb7449d4640 1 -- 172.21.15.38:0/1481435460 shutdown_connections 2024-03-26T06:37:17.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.008+0000 7fb7449d4640 1 --2- 172.21.15.38:0/1481435460 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb740102eb0 0x7fb740109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.008+0000 7fb7449d4640 1 --2- 172.21.15.38:0/1481435460 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7401024f0 0x7fb740102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.008+0000 7fb7449d4640 1 --2- 172.21.15.38:0/1481435460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb740101300 0x7fb740101720 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.013 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.008+0000 7fb7449d4640 1 -- 172.21.15.38:0/1481435460 >> 172.21.15.38:0/1481435460 conn(0x7fb7400fc890 msgr2=0x7fb7400fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.008+0000 7fb7449d4640 1 -- 172.21.15.38:0/1481435460 shutdown_connections 2024-03-26T06:37:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.008+0000 7fb7449d4640 1 -- 172.21.15.38:0/1481435460 wait complete. 2024-03-26T06:37:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb7449d4640 1 Processor -- start 2024-03-26T06:37:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb7449d4640 1 -- start start 2024-03-26T06:37:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb7449d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb740101300 0x7fb740197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb7449d4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7401024f0 0x7fb740197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb7449d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb740102eb0 0x7fb74019ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb7449d4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74010cfe0 con 0x7fb740101300 2024-03-26T06:37:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb7449d4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74010ce60 con 0x7fb7401024f0 2024-03-26T06:37:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb7449d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74010d160 con 0x7fb740102eb0 2024-03-26T06:37:17.014 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb73e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb740101300 0x7fb740197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb73e59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb740101300 0x7fb740197670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42132/0 (socket says 172.21.15.38:42132) 2024-03-26T06:37:17.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb73e59c640 1 -- 172.21.15.38:0/1664750815 learned_addr learned my addr 172.21.15.38:0/1664750815 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:17.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb737fff640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7401024f0 0x7fb740197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb73ed9d640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb740102eb0 0x7fb74019ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb73e59c640 1 -- 172.21.15.38:0/1664750815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb740102eb0 msgr2=0x7fb74019ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb73e59c640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb740102eb0 0x7fb74019ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb73e59c640 1 -- 172.21.15.38:0/1664750815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7401024f0 msgr2=0x7fb740197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb73e59c640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7401024f0 0x7fb740197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb73e59c640 1 -- 172.21.15.38:0/1664750815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb74019f290 con 0x7fb740101300 2024-03-26T06:37:17.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.009+0000 7fb737fff640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7401024f0 0x7fb740197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:37:17.015 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.010+0000 7fb73e59c640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb740101300 0x7fb740197670 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb730000c00 tx=0x7fb730005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:17.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.010+0000 7fb7377fe640 1 -- 172.21.15.38:0/1664750815 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730043020 con 0x7fb740101300 2024-03-26T06:37:17.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.010+0000 7fb7377fe640 1 -- 172.21.15.38:0/1664750815 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7300054e0 con 0x7fb740101300 2024-03-26T06:37:17.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.010+0000 7fb7449d4640 1 -- 172.21.15.38:0/1664750815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74019f520 con 0x7fb740101300 2024-03-26T06:37:17.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.010+0000 7fb7377fe640 1 -- 172.21.15.38:0/1664750815 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb730003e60 con 0x7fb740101300 2024-03-26T06:37:17.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.010+0000 7fb7449d4640 1 -- 172.21.15.38:0/1664750815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74019fa70 con 0x7fb740101300 2024-03-26T06:37:17.016 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.011+0000 7fb7449d4640 1 -- 172.21.15.38:0/1664750815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb740108620 con 0x7fb740101300 2024-03-26T06:37:17.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.012+0000 7fb7377fe640 1 -- 172.21.15.38:0/1664750815 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb730046020 con 0x7fb740101300 2024-03-26T06:37:17.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.013+0000 7fb7377fe640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb72c0a2800 0x7fb72c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.013+0000 7fb737fff640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb72c0a2800 0x7fb72c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.013+0000 7fb7377fe640 1 -- 172.21.15.38:0/1664750815 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fb7300f0d30 con 0x7fb740101300 2024-03-26T06:37:17.017 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.013+0000 7fb737fff640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb72c0a2800 0x7fb72c0a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb740198b80 tx=0x7fb728005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:17.019 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.017+0000 7fb7377fe640 1 -- 172.21.15.38:0/1664750815 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb73004d020 con 0x7fb740101300 2024-03-26T06:37:17.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.201+0000 7fb7449d4640 1 -- 172.21.15.38:0/1664750815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb74002d0b0 con 0x7fb740101300 2024-03-26T06:37:17.204 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.202+0000 7fb7377fe640 1 -- 172.21.15.38:0/1664750815 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb7300bddd0 con 0x7fb740101300 2024-03-26T06:37:17.204 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-03-26T06:37:17.206 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 -- 172.21.15.38:0/1664750815 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb72c0a2800 msgr2=0x7fb72c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb72c0a2800 0x7fb72c0a4cc0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb740198b80 tx=0x7fb728005ec0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 -- 172.21.15.38:0/1664750815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb740101300 msgr2=0x7fb740197670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb740101300 0x7fb740197670 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb730000c00 tx=0x7fb730005dd0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 -- 172.21.15.38:0/1664750815 shutdown_connections 2024-03-26T06:37:17.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fb72c0a2800 0x7fb72c0a4cc0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb740102eb0 0x7fb74019ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb7401024f0 0x7fb740197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 --2- 172.21.15.38:0/1664750815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb740101300 0x7fb740197670 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 -- 172.21.15.38:0/1664750815 >> 172.21.15.38:0/1664750815 conn(0x7fb7400fc890 msgr2=0x7fb7401057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:17.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 -- 172.21.15.38:0/1664750815 shutdown_connections 2024-03-26T06:37:17.207 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.204+0000 7fb7449d4640 1 -- 172.21.15.38:0/1664750815 wait complete. 2024-03-26T06:37:17.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-26T06:37:17.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.321+0000 7fef92206640 1 -- 172.21.15.38:0/1690577127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef8c101db0 msgr2=0x7fef8c108640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.321+0000 7fef92206640 1 --2- 172.21.15.38:0/1690577127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef8c101db0 0x7fef8c108640 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fef84009ca0 tx=0x7fef8402c0f0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.321+0000 7fef92206640 1 -- 172.21.15.38:0/1690577127 shutdown_connections 2024-03-26T06:37:17.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.321+0000 7fef92206640 1 --2- 172.21.15.38:0/1690577127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef8c101db0 0x7fef8c108640 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.321+0000 7fef92206640 1 --2- 172.21.15.38:0/1690577127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef8c1013f0 0x7fef8c101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.321+0000 7fef92206640 1 --2- 172.21.15.38:0/1690577127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef8c100200 0x7fef8c100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.321+0000 7fef92206640 1 -- 172.21.15.38:0/1690577127 >> 172.21.15.38:0/1690577127 conn(0x7fef8c0fb790 msgr2=0x7fef8c0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:17.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.321+0000 7fef92206640 1 -- 172.21.15.38:0/1690577127 shutdown_connections 2024-03-26T06:37:17.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.321+0000 7fef92206640 1 -- 172.21.15.38:0/1690577127 wait complete. 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.321+0000 7fef92206640 1 Processor -- start 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.321+0000 7fef92206640 1 -- start start 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef92206640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef8c100200 0x7fef8c104190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef92206640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef8c1013f0 0x7fef8c1027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef92206640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef8c101db0 0x7fef8c102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef92206640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c10be90 con 0x7fef8c1013f0 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef92206640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c10bd10 con 0x7fef8c101db0 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef92206640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef8c10c010 con 0x7fef8c100200 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef8c1013f0 0x7fef8c1027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef8c1013f0 0x7fef8c1027d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42160/0 (socket says 172.21.15.38:42160) 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8affd640 1 -- 172.21.15.38:0/1021284481 learned_addr learned my addr 172.21.15.38:0/1021284481 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8b7fe640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef8c100200 0x7fef8c104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8bfff640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef8c101db0 0x7fef8c102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8affd640 1 -- 172.21.15.38:0/1021284481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef8c100200 msgr2=0x7fef8c104190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8affd640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef8c100200 0x7fef8c104190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8affd640 1 -- 172.21.15.38:0/1021284481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef8c101db0 msgr2=0x7fef8c102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8affd640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef8c101db0 0x7fef8c102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8affd640 1 -- 172.21.15.38:0/1021284481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c1035a0 con 0x7fef8c1013f0 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8bfff640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef8c101db0 0x7fef8c102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef8affd640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef8c1013f0 0x7fef8c1027d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fef8000c970 tx=0x7fef8000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef88ff9640 1 -- 172.21.15.38:0/1021284481 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef80007bb0 con 0x7fef8c1013f0 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef92206640 1 -- 172.21.15.38:0/1021284481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef8c1a16f0 con 0x7fef8c1013f0 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef88ff9640 1 -- 172.21.15.38:0/1021284481 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef80007d50 con 0x7fef8c1013f0 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef88ff9640 1 -- 172.21.15.38:0/1021284481 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8000f450 con 0x7fef8c1013f0 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.322+0000 7fef92206640 1 -- 172.21.15.38:0/1021284481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef8c1a1bf0 con 0x7fef8c1013f0 2024-03-26T06:37:17.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.323+0000 7fef92206640 1 -- 172.21.15.38:0/1021284481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef8c1a1f10 con 0x7fef8c1013f0 2024-03-26T06:37:17.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.324+0000 7fef88ff9640 1 -- 172.21.15.38:0/1021284481 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fef80005020 con 0x7fef8c1013f0 2024-03-26T06:37:17.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.325+0000 7fef88ff9640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fef740a2800 0x7fef740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.325+0000 7fef8b7fe640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fef740a2800 0x7fef740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.325+0000 7fef88ff9640 1 -- 172.21.15.38:0/1021284481 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fef800ce900 con 0x7fef8c1013f0 2024-03-26T06:37:17.330 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.325+0000 7fef8b7fe640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fef740a2800 0x7fef740a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fef8c101250 tx=0x7fef7c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:17.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.329+0000 7fef88ff9640 1 -- 172.21.15.38:0/1021284481 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fef80014030 con 0x7fef8c1013f0 2024-03-26T06:37:17.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.511+0000 7fef92206640 1 -- 172.21.15.38:0/1021284481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fef8c1a20e0 con 0x7fef8c1013f0 2024-03-26T06:37:17.514 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.511+0000 7fef88ff9640 1 -- 172.21.15.38:0/1021284481 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7fef8009b9a0 con 0x7fef8c1013f0 2024-03-26T06:37:17.514 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-03-26T06:37:17.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.513+0000 7fef92206640 1 -- 172.21.15.38:0/1021284481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fef740a2800 msgr2=0x7fef740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.513+0000 7fef92206640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fef740a2800 0x7fef740a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fef8c101250 tx=0x7fef7c00b390 comp rx=0 tx=0).stop 2024-03-26T06:37:17.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.513+0000 7fef92206640 1 -- 172.21.15.38:0/1021284481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef8c1013f0 msgr2=0x7fef8c1027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.516 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.513+0000 7fef92206640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef8c1013f0 0x7fef8c1027d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fef8000c970 tx=0x7fef8000ce30 comp rx=0 tx=0).stop 2024-03-26T06:37:17.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.513+0000 7fef92206640 1 -- 172.21.15.38:0/1021284481 shutdown_connections 2024-03-26T06:37:17.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.513+0000 7fef92206640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fef740a2800 0x7fef740a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.513+0000 7fef92206640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fef8c101db0 0x7fef8c102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.513+0000 7fef92206640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fef8c1013f0 0x7fef8c1027d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.513+0000 7fef92206640 1 --2- 172.21.15.38:0/1021284481 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fef8c100200 0x7fef8c104190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.513+0000 7fef92206640 1 -- 172.21.15.38:0/1021284481 >> 172.21.15.38:0/1021284481 conn(0x7fef8c0fb790 msgr2=0x7fef8c0fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:17.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.514+0000 7fef92206640 1 -- 172.21.15.38:0/1021284481 shutdown_connections 2024-03-26T06:37:17.517 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.514+0000 7fef92206640 1 -- 172.21.15.38:0/1021284481 wait complete. 2024-03-26T06:37:17.526 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:17 smithi038 ceph-mon[32265]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:37:17.526 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:17 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3612921934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:37:17.526 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:17 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3107079890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:37:17.526 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:17 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1664750815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:37:17.527 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:37:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.627+0000 7fcd36b09640 1 -- 172.21.15.38:0/2612491531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd30101e10 msgr2=0x7fcd30104200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.627+0000 7fcd36b09640 1 --2- 172.21.15.38:0/2612491531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd30101e10 0x7fcd30104200 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fcd28009ca0 tx=0x7fcd2802c0f0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.627+0000 7fcd36b09640 1 -- 172.21.15.38:0/2612491531 shutdown_connections 2024-03-26T06:37:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.627+0000 7fcd36b09640 1 --2- 172.21.15.38:0/2612491531 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd30107070 0x7fcd301094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.627+0000 7fcd36b09640 1 --2- 172.21.15.38:0/2612491531 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd30104740 0x7fcd30106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.627+0000 7fcd36b09640 1 --2- 172.21.15.38:0/2612491531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd30101e10 0x7fcd30104200 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.627+0000 7fcd36b09640 1 -- 172.21.15.38:0/2612491531 >> 172.21.15.38:0/2612491531 conn(0x7fcd300fb820 msgr2=0x7fcd300fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:17.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.627+0000 7fcd36b09640 1 -- 172.21.15.38:0/2612491531 shutdown_connections 2024-03-26T06:37:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.627+0000 7fcd36b09640 1 -- 172.21.15.38:0/2612491531 wait complete. 2024-03-26T06:37:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd36b09640 1 Processor -- start 2024-03-26T06:37:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd36b09640 1 -- start start 2024-03-26T06:37:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd36b09640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd30101e10 0x7fcd301975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd36b09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd30104740 0x7fcd30197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.634 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd36b09640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd30107070 0x7fcd3019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd36b09640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3010cfb0 con 0x7fcd30104740 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd36b09640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3010ce30 con 0x7fcd30101e10 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd36b09640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3010d130 con 0x7fcd30107070 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd2ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd30104740 0x7fcd30197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd2ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd30104740 0x7fcd30197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42188/0 (socket says 172.21.15.38:42188) 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd2ffff640 1 -- 172.21.15.38:0/3080728784 learned_addr learned my addr 172.21.15.38:0/3080728784 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd348a5640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd30101e10 0x7fcd301975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd350a6640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd30107070 0x7fcd3019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd2ffff640 1 -- 172.21.15.38:0/3080728784 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd30107070 msgr2=0x7fcd3019eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd2ffff640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd30107070 0x7fcd3019eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd2ffff640 1 -- 172.21.15.38:0/3080728784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd30101e10 msgr2=0x7fcd301975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd2ffff640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd30101e10 0x7fcd301975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.635 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd2ffff640 1 -- 172.21.15.38:0/3080728784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3019f1c0 con 0x7fcd30104740 2024-03-26T06:37:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd350a6640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd30107070 0x7fcd3019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd348a5640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd30101e10 0x7fcd301975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.628+0000 7fcd2ffff640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd30104740 0x7fcd30197ae0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000c970 tx=0x7fcd2000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.629+0000 7fcd2dffb640 1 -- 172.21.15.38:0/3080728784 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd20007bb0 con 0x7fcd30104740 2024-03-26T06:37:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.629+0000 7fcd2dffb640 1 -- 172.21.15.38:0/3080728784 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd20007d50 con 0x7fcd30104740 2024-03-26T06:37:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.629+0000 7fcd36b09640 1 -- 172.21.15.38:0/3080728784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3019f4b0 con 0x7fcd30104740 2024-03-26T06:37:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.629+0000 7fcd2dffb640 1 -- 172.21.15.38:0/3080728784 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2000f450 con 0x7fcd30104740 2024-03-26T06:37:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.629+0000 7fcd36b09640 1 -- 172.21.15.38:0/3080728784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3019fa50 con 0x7fcd30104740 2024-03-26T06:37:17.636 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.630+0000 7fcd36b09640 1 -- 172.21.15.38:0/3080728784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd300fd310 con 0x7fcd30104740 2024-03-26T06:37:17.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.631+0000 7fcd2dffb640 1 -- 172.21.15.38:0/3080728784 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcd200046c0 con 0x7fcd30104740 2024-03-26T06:37:17.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.631+0000 7fcd2dffb640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcd1c0a2880 0x7fcd1c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.632+0000 7fcd348a5640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcd1c0a2880 0x7fcd1c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.632+0000 7fcd2dffb640 1 -- 172.21.15.38:0/3080728784 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fcd200ceae0 con 0x7fcd30104740 2024-03-26T06:37:17.637 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.632+0000 7fcd348a5640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcd1c0a2880 0x7fcd1c0a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcd28006010 tx=0x7fcd28005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:17.638 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.636+0000 7fcd2dffb640 1 -- 172.21.15.38:0/3080728784 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcd20014030 con 0x7fcd30104740 2024-03-26T06:37:17.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.862+0000 7fcd36b09640 1 -- 172.21.15.38:0/3080728784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd3002ce70 con 0x7fcd30104740 2024-03-26T06:37:17.865 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.863+0000 7fcd2dffb640 1 -- 172.21.15.38:0/3080728784 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+11346 (secure 0 0 0) 0x7fcd2009bb80 con 0x7fcd30104740 2024-03-26T06:37:17.866 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:37:17.866 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/357411663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":357411663},{"type":"v1","addr":"172.21.15.38:6839","nonce":357411663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/654319512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":654319512},{"type":"v1","addr":"172.21.15.38:6841","nonce":654319512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":22,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-26T06:35:26.477444+0000","modified":"2024-03-26T06:35:32.167599+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24502,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24472":{"gid":24472,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/2484220351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":2484220351},{"type":"v1","addr":"172.21.15.170:6837","nonce":2484220351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/359437185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":359437185},{"type":"v1","addr":"172.21.15.121:6839","nonce":359437185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-26T06:37:17.866 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 22 2024-03-26T06:37:17.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 -- 172.21.15.38:0/3080728784 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcd1c0a2880 msgr2=0x7fcd1c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcd1c0a2880 0x7fcd1c0a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fcd28006010 tx=0x7fcd28005c50 comp rx=0 tx=0).stop 2024-03-26T06:37:17.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 -- 172.21.15.38:0/3080728784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd30104740 msgr2=0x7fcd30197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.868 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd30104740 0x7fcd30197ae0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fcd2000c970 tx=0x7fcd2000ce30 comp rx=0 tx=0).stop 2024-03-26T06:37:17.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 -- 172.21.15.38:0/3080728784 shutdown_connections 2024-03-26T06:37:17.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fcd1c0a2880 0x7fcd1c0a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcd30107070 0x7fcd3019eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcd30104740 0x7fcd30197ae0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 --2- 172.21.15.38:0/3080728784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd30101e10 0x7fcd301975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 -- 172.21.15.38:0/3080728784 >> 172.21.15.38:0/3080728784 conn(0x7fcd300fb820 msgr2=0x7fcd30102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:17.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 -- 172.21.15.38:0/3080728784 shutdown_connections 2024-03-26T06:37:17.869 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.865+0000 7fcd36b09640 1 -- 172.21.15.38:0/3080728784 wait complete. 2024-03-26T06:37:17.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:37:17.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:17 smithi121 ceph-mon[36349]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:37:17.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:17 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3612921934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:37:17.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:17 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3107079890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:37:17.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:17 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1664750815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:37:17.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:17 smithi170 ceph-mon[36912]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:37:17.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:17 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3612921934' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:37:17.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:17 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3107079890' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:37:17.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:17 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1664750815' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:37:17.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.979+0000 7fede5b37640 1 -- 172.21.15.38:0/589894343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0104740 msgr2=0x7fede0106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.984 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.979+0000 7fede5b37640 1 --2- 172.21.15.38:0/589894343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0104740 0x7fede0106b30 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fedd4009900 tx=0x7fedd402bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.979+0000 7fede5b37640 1 -- 172.21.15.38:0/589894343 shutdown_connections 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.979+0000 7fede5b37640 1 --2- 172.21.15.38:0/589894343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0107070 0x7fede01094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.979+0000 7fede5b37640 1 --2- 172.21.15.38:0/589894343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0104740 0x7fede0106b30 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.979+0000 7fede5b37640 1 --2- 172.21.15.38:0/589894343 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fede0101e10 0x7fede0104200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.979+0000 7fede5b37640 1 -- 172.21.15.38:0/589894343 >> 172.21.15.38:0/589894343 conn(0x7fede00fb820 msgr2=0x7fede00fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.980+0000 7fede5b37640 1 -- 172.21.15.38:0/589894343 shutdown_connections 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.980+0000 7fede5b37640 1 -- 172.21.15.38:0/589894343 wait complete. 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.980+0000 7fede5b37640 1 Processor -- start 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.980+0000 7fede5b37640 1 -- start start 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.980+0000 7fede5b37640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0101e10 0x7fede0197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.980+0000 7fede5b37640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0104740 0x7fede0197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.985 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.980+0000 7fede5b37640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fede0107070 0x7fede019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.980+0000 7fede5b37640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede010cfb0 con 0x7fede0104740 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.980+0000 7fede5b37640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede010ce30 con 0x7fede0101e10 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.980+0000 7fede5b37640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fede010d130 con 0x7fede0107070 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0104740 0x7fede0197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddeffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0104740 0x7fede0197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42206/0 (socket says 172.21.15.38:42206) 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddeffd640 1 -- 172.21.15.38:0/1888587771 learned_addr learned my addr 172.21.15.38:0/1888587771 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddffff640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fede0107070 0x7fede019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddf7fe640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0101e10 0x7fede0197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddeffd640 1 -- 172.21.15.38:0/1888587771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fede0107070 msgr2=0x7fede019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddeffd640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fede0107070 0x7fede019ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddeffd640 1 -- 172.21.15.38:0/1888587771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0101e10 msgr2=0x7fede0197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddeffd640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0101e10 0x7fede0197640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:17.986 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddeffd640 1 -- 172.21.15.38:0/1888587771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede019f260 con 0x7fede0104740 2024-03-26T06:37:17.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddf7fe640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0101e10 0x7fede0197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:37:17.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddeffd640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0104740 0x7fede0197b80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fedd4000c00 tx=0x7fedd4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:17.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddcff9640 1 -- 172.21.15.38:0/1888587771 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4041070 con 0x7fede0104740 2024-03-26T06:37:17.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7fede5b37640 1 -- 172.21.15.38:0/1888587771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fede019f4f0 con 0x7fede0104740 2024-03-26T06:37:17.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddcff9640 1 -- 172.21.15.38:0/1888587771 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedd40044d0 con 0x7fede0104740 2024-03-26T06:37:17.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7feddcff9640 1 -- 172.21.15.38:0/1888587771 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd4034440 con 0x7fede0104740 2024-03-26T06:37:17.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.981+0000 7fede5b37640 1 -- 172.21.15.38:0/1888587771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fede019f9e0 con 0x7fede0104740 2024-03-26T06:37:17.987 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.982+0000 7fede5b37640 1 -- 172.21.15.38:0/1888587771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede00fd310 con 0x7fede0104740 2024-03-26T06:37:17.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.983+0000 7feddcff9640 1 -- 172.21.15.38:0/1888587771 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fedd4044070 con 0x7fede0104740 2024-03-26T06:37:17.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.984+0000 7feddcff9640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedc80a2800 0x7fedc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:17.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.984+0000 7feddf7fe640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedc80a2800 0x7fedc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:17.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.984+0000 7feddcff9640 1 -- 172.21.15.38:0/1888587771 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fedd40f0600 con 0x7fede0104740 2024-03-26T06:37:17.988 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.984+0000 7feddf7fe640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedc80a2800 0x7fedc80a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009c20 tx=0x7fedd0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:17.990 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:17.988+0000 7feddcff9640 1 -- 172.21.15.38:0/1888587771 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fedd40bd6a0 con 0x7fede0104740 2024-03-26T06:37:18.173 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.171+0000 7fede5b37640 1 -- 172.21.15.38:0/1888587771 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fede0061ef0 con 0x7fede0104740 2024-03-26T06:37:18.175 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.172+0000 7feddcff9640 1 -- 172.21.15.38:0/1888587771 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22634 (secure 0 0 0) 0x7fedd40bcfa0 con 0x7fede0104740 2024-03-26T06:37:18.175 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:37:18.175 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":88,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:36:27.512655+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:37:18.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 -- 172.21.15.38:0/1888587771 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedc80a2800 msgr2=0x7fedc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedc80a2800 0x7fedc80a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009c20 tx=0x7fedd0005f90 comp rx=0 tx=0).stop 2024-03-26T06:37:18.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 -- 172.21.15.38:0/1888587771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0104740 msgr2=0x7fede0197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0104740 0x7fede0197b80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fedd4000c00 tx=0x7fedd4005d50 comp rx=0 tx=0).stop 2024-03-26T06:37:18.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 -- 172.21.15.38:0/1888587771 shutdown_connections 2024-03-26T06:37:18.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fedc80a2800 0x7fedc80a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fede0107070 0x7fede019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.177 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fede0104740 0x7fede0197b80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 --2- 172.21.15.38:0/1888587771 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fede0101e10 0x7fede0197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 -- 172.21.15.38:0/1888587771 >> 172.21.15.38:0/1888587771 conn(0x7fede00fb820 msgr2=0x7fede0102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 -- 172.21.15.38:0/1888587771 shutdown_connections 2024-03-26T06:37:18.178 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.174+0000 7fede5b37640 1 -- 172.21.15.38:0/1888587771 wait complete. 2024-03-26T06:37:18.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-26T06:35:26.477444+0000', 'modified': '2024-03-26T06:35:32.167599+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24539, 'mds_1': 24502, 'mds_2': 24488}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24460': {'gid': 24460, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6835/2374885191', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6834', 'nonce': 2374885191}, {'type': 'v1', 'addr': '172.21.15.170:6835', 'nonce': 2374885191}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24464': {'gid': 24464, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.121:6835/3384748966', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6834', 'nonce': 3384748966}, {'type': 'v1', 'addr': '172.21.15.121:6835', 'nonce': 3384748966}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24472': {'gid': 24472, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.170:6837/2484220351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6836', 'nonce': 2484220351}, {'type': 'v1', 'addr': '172.21.15.170:6837', 'nonce': 2484220351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24488': {'gid': 24488, 'name': 'h', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.121:6837/1114423776', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6836', 'nonce': 1114423776}, {'type': 'v1', 'addr': '172.21.15.121:6837', 'nonce': 1114423776}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24502': {'gid': 24502, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.170:6839/2204378068', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.170:6838', 'nonce': 2204378068}, {'type': 'v1', 'addr': '172.21.15.170:6839', 'nonce': 2204378068}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24539': {'gid': 24539, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.121:6839/359437185', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.121:6838', 'nonce': 359437185}, {'type': 'v1', 'addr': '172.21.15.121:6839', 'nonce': 359437185}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-26T06:37:18.189 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-26T06:37:18.189 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-26T06:37:18.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.289+0000 7fde1cebf640 1 -- 172.21.15.38:0/3835063947 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde18101400 msgr2=0x7fde18101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.294 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.289+0000 7fde1cebf640 1 --2- 172.21.15.38:0/3835063947 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde18101400 0x7fde18101880 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fde00009900 tx=0x7fde0002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:18.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 -- 172.21.15.38:0/3835063947 shutdown_connections 2024-03-26T06:37:18.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 --2- 172.21.15.38:0/3835063947 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde18101dc0 0x7fde18108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 --2- 172.21.15.38:0/3835063947 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde18101400 0x7fde18101880 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 --2- 172.21.15.38:0/3835063947 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde18100210 0x7fde18100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 -- 172.21.15.38:0/3835063947 >> 172.21.15.38:0/3835063947 conn(0x7fde180fb7a0 msgr2=0x7fde180fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 -- 172.21.15.38:0/3835063947 shutdown_connections 2024-03-26T06:37:18.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 -- 172.21.15.38:0/3835063947 wait complete. 2024-03-26T06:37:18.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 Processor -- start 2024-03-26T06:37:18.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 -- start start 2024-03-26T06:37:18.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde18100210 0x7fde1806a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.295 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde18101400 0x7fde1819c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde18101dc0 0x7fde1819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1810bea0 con 0x7fde18101dc0 2024-03-26T06:37:18.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.290+0000 7fde1cebf640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1810bd20 con 0x7fde18100210 2024-03-26T06:37:18.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde1cebf640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1810c020 con 0x7fde18101400 2024-03-26T06:37:18.296 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde16d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde18101dc0 0x7fde1819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde16d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde18101dc0 0x7fde1819e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42222/0 (socket says 172.21.15.38:42222) 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde16d9d640 1 -- 172.21.15.38:0/2147123491 learned_addr learned my addr 172.21.15.38:0/2147123491 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde0ffff640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde18101400 0x7fde1819c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde1659c640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde18100210 0x7fde1806a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde16d9d640 1 -- 172.21.15.38:0/2147123491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde18101400 msgr2=0x7fde1819c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde16d9d640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde18101400 0x7fde1819c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde16d9d640 1 -- 172.21.15.38:0/2147123491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde18100210 msgr2=0x7fde1806a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde16d9d640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde18100210 0x7fde1806a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde16d9d640 1 -- 172.21.15.38:0/2147123491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1819f260 con 0x7fde18101dc0 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde16d9d640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde18101dc0 0x7fde1819e9a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fde1000cad0 tx=0x7fde1000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde0f7fe640 1 -- 172.21.15.38:0/2147123491 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1000ae20 con 0x7fde18101dc0 2024-03-26T06:37:18.297 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde0f7fe640 1 -- 172.21.15.38:0/2147123491 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde10005020 con 0x7fde18101dc0 2024-03-26T06:37:18.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde1cebf640 1 -- 172.21.15.38:0/2147123491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1819f550 con 0x7fde18101dc0 2024-03-26T06:37:18.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde0f7fe640 1 -- 172.21.15.38:0/2147123491 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1000f6f0 con 0x7fde18101dc0 2024-03-26T06:37:18.298 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.291+0000 7fde1cebf640 1 -- 172.21.15.38:0/2147123491 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1819fa20 con 0x7fde18101dc0 2024-03-26T06:37:18.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde0f7fe640 1 -- 172.21.15.38:0/2147123491 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fde100051d0 con 0x7fde18101dc0 2024-03-26T06:37:18.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde1fb76740 1 Processor -- start 2024-03-26T06:37:18.305 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde1fb76740 1 -- start start 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc958390e0 0x55fc95839500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc9583e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc9583ec40 0x55fc958410d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde1fb76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9579e7c0 con 0x55fc95839ad0 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde1fb76740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc95747b50 con 0x55fc958390e0 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde1fb76740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9579a860 con 0x55fc9583ec40 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde1659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc9583e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde0f7fe640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fde040a2800 0x7fde040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde1659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc9583e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42226/0 (socket says 172.21.15.38:42226) 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.294+0000 7fde1659c640 1 -- 172.21.15.38:0/3769131955 learned_addr learned my addr 172.21.15.38:0/3769131955 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde16d9d640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fde040a2800 0x7fde040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde0ffff640 1 --2- 172.21.15.38:0/3769131955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc9583ec40 0x55fc958410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.306 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde0ffff640 1 --2- 172.21.15.38:0/3769131955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc958390e0 0x55fc95839500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde1659c640 1 -- 172.21.15.38:0/3769131955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc9583ec40 msgr2=0x55fc958410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde1659c640 1 --2- 172.21.15.38:0/3769131955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc9583ec40 0x55fc958410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde1659c640 1 -- 172.21.15.38:0/3769131955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc958390e0 msgr2=0x55fc95839500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde1659c640 1 --2- 172.21.15.38:0/3769131955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc958390e0 0x55fc95839500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde0f7fe640 1 -- 172.21.15.38:0/2147123491 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fde100cead0 con 0x7fde18101dc0 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde1659c640 1 -- 172.21.15.38:0/3769131955 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc9575b9a0 con 0x55fc95839ad0 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde0ffff640 1 --2- 172.21.15.38:0/3769131955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc958390e0 0x55fc95839500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde1659c640 1 --2- 172.21.15.38:0/3769131955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc9583e5d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fde0800baa0 tx=0x7fde0800bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e02451dd606b51b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde16d9d640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fde040a2800 0x7fde040a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fde1006e170 tx=0x7fde100c5ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde0cff9640 1 -- 172.21.15.38:0/3769131955 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde08017420 con 0x55fc95839ad0 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde0cff9640 1 -- 172.21.15.38:0/3769131955 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde08017aa0 con 0x55fc95839ad0 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.295+0000 7fde0cff9640 1 -- 172.21.15.38:0/3769131955 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0800fb20 con 0x55fc95839ad0 2024-03-26T06:37:18.307 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 -- 172.21.15.38:0/3769131955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 msgr2=0x55fc9583e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 --2- 172.21.15.38:0/3769131955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc9583e5d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fde0800baa0 tx=0x7fde0800bf60 comp rx=0 tx=0).stop 2024-03-26T06:37:18.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 -- 172.21.15.38:0/3769131955 shutdown_connections 2024-03-26T06:37:18.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 --2- 172.21.15.38:0/3769131955 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc9583ec40 0x55fc958410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 --2- 172.21.15.38:0/3769131955 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc9583e5d0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 --2- 172.21.15.38:0/3769131955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc958390e0 0x55fc95839500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 -- 172.21.15.38:0/3769131955 >> 172.21.15.38:0/3769131955 conn(0x55fc95801400 msgr2=0x55fc95834a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 -- 172.21.15.38:0/3769131955 shutdown_connections 2024-03-26T06:37:18.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 -- 172.21.15.38:0/3769131955 wait complete. 2024-03-26T06:37:18.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 -- start start 2024-03-26T06:37:18.308 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 Processor -- start 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.296+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc958390e0 0x55fc95843560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc95839ad0 0x55fc95843aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc9584aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde1fb76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc95841bf0 con 0x55fc9583ec40 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde1fb76740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9575af90 con 0x55fc95839ad0 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde1fb76740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc95841d70 con 0x55fc958390e0 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde1fb76740 10 client.0 _ll_register_callbacks cb 0x55fc95796400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde0ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc9584aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde0ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc9584aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42230/0 (socket says 172.21.15.38:42230) 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde0ffff640 1 -- 172.21.15.38:0/2022310990 learned_addr learned my addr 172.21.15.38:0/2022310990 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde0ffff640 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc958390e0 0x55fc95843560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde0ffff640 1 -- 172.21.15.38:0/2022310990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc958390e0 msgr2=0x55fc95843560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.309 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde0ffff640 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc958390e0 0x55fc95843560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde0ffff640 1 -- 172.21.15.38:0/2022310990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc95839ad0 msgr2=0x55fc95843aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde1659c640 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc95839ad0 0x55fc95843aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde0ffff640 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc95839ad0 0x55fc95843aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde0ffff640 1 -- 172.21.15.38:0/2022310990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc954b32d0 con 0x55fc9583ec40 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde0ffff640 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc9584aae0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fde0002cfd0 tx=0x7fde0002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=959e06c71d282f89 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fddfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fddfdffb640 1 -- 172.21.15.38:0/2022310990 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde000053c0 con 0x55fc9583ec40 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fddfdffb640 1 -- 172.21.15.38:0/2022310990 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fde00005560 con 0x55fc9583ec40 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde1fb76740 1 -- 172.21.15.38:0/2022310990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde08017aa0 con 0x55fc9583ec40 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fddfdffb640 1 -- 172.21.15.38:0/2022310990 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde00053050 con 0x55fc9583ec40 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde1fb76740 10 client.15216 fetch_fsmap learned FSMap version 22 2024-03-26T06:37:18.310 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.297+0000 7fde1fb76740 1 -- 172.21.15.38:0/2022310990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55fc9557d6b0 con 0x55fc9583ec40 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.298+0000 7fddfdffb640 1 -- 172.21.15.38:0/2022310990 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde08017aa0 con 0x55fc9583ec40 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.298+0000 7fddfdffb640 1 -- 172.21.15.38:0/2022310990 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fde00003bc0 con 0x55fc9583ec40 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.298+0000 7fde1fb76740 10 client.15216 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.298+0000 7fde1fb76740 10 client.15216 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.298+0000 7fde1fb76740 20 client.15216 populate_metadata read hostname 'smithi038' 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.298+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x55fc95850020 0x55fc9584f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.298+0000 7fde1fb76740 4 client.15216 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.298+0000 7fde1fb76740 1 -- 172.21.15.38:0/2022310990 --> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fc957abd00 con 0x55fc95850020 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.298+0000 7fde1659c640 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x55fc95850020 0x55fc9584f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fddfdffb640 10 client.15216 ms_handle_connect on v2:172.21.15.121:6836/1114423776 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1659c640 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x55fc95850020 0x55fc9584f0d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8163439532491855 server_cookie=38ab76ca9f5af51c in_seq=0 out_seq=0 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fddfdffb640 1 -- 172.21.15.38:0/2022310990 <== mds.2 v2:172.21.15.121:6836/1114423776 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fc957abd00 con 0x55fc95850020 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fddfdffb640 10 client.15216 handle_command_reply: tid=0 2024-03-26T06:37:18.311 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1fb76740 1 client.15216 shutdown 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1fb76740 1 -- 172.21.15.38:0/2022310990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 msgr2=0x55fc9584aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc9584aae0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fde0002cfd0 tx=0x7fde0002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1fb76740 1 -- 172.21.15.38:0/2022310990 shutdown_connections 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x55fc95850020 0x55fc9584f0d0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc9584aae0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc95839ad0 0x55fc95843aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fddfdffb640 0 client.15216 ms_handle_reset on v2:172.21.15.121:6836/1114423776 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2022310990 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc958390e0 0x55fc95843560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1fb76740 1 -- 172.21.15.38:0/2022310990 >> 172.21.15.38:0/2022310990 conn(0x55fc95801400 msgr2=0x55fc95833c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1fb76740 1 -- 172.21.15.38:0/2022310990 shutdown_connections 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.299+0000 7fde1fb76740 1 -- 172.21.15.38:0/2022310990 wait complete. 2024-03-26T06:37:18.312 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.300+0000 7fde1fb76740 20 client.15216 trim_cache size 0 max 16384 2024-03-26T06:37:18.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.315+0000 7fde1fb76740 1 Processor -- start 2024-03-26T06:37:18.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.315+0000 7fde1fb76740 1 -- start start 2024-03-26T06:37:18.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.315+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc958390e0 0x55fc95842140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.315+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc95839ad0 0x55fc958584c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.315+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc95843560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.315+0000 7fde1fb76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9584f840 con 0x55fc9583ec40 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.315+0000 7fde1fb76740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc9584fea0 con 0x55fc958390e0 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.315+0000 7fde1fb76740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc95850500 con 0x55fc95839ad0 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.315+0000 7fde0ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc95843560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.315+0000 7fde0ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc95843560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42234/0 (socket says 172.21.15.38:42234) 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.315+0000 7fde0ffff640 1 -- 172.21.15.38:0/1976632700 learned_addr learned my addr 172.21.15.38:0/1976632700 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.316+0000 7fde0ffff640 1 -- 172.21.15.38:0/1976632700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc95839ad0 msgr2=0x55fc958584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.316+0000 7fde1659c640 1 --2- 172.21.15.38:0/1976632700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc95839ad0 0x55fc958584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.316+0000 7fde0ffff640 1 --2- 172.21.15.38:0/1976632700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc95839ad0 0x55fc958584c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.316+0000 7fde0ffff640 1 -- 172.21.15.38:0/1976632700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc958390e0 msgr2=0x55fc95842140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.316+0000 7fde0ffff640 1 --2- 172.21.15.38:0/1976632700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc958390e0 0x55fc95842140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.316+0000 7fde0ffff640 1 -- 172.21.15.38:0/1976632700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc9583e1e0 con 0x55fc9583ec40 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.316+0000 7fde0ffff640 1 --2- 172.21.15.38:0/1976632700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc95843560 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fde00001eb0 tx=0x7fde00005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64bf53d087339b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.316+0000 7fddfffff640 1 -- 172.21.15.38:0/1976632700 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde000053c0 con 0x55fc9583ec40 2024-03-26T06:37:18.324 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.316+0000 7fddfffff640 1 -- 172.21.15.38:0/1976632700 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde00004470 con 0x55fc9583ec40 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.316+0000 7fddfffff640 1 -- 172.21.15.38:0/1976632700 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde00034ad0 con 0x55fc9583ec40 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 -- 172.21.15.38:0/1976632700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 msgr2=0x55fc95843560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 --2- 172.21.15.38:0/1976632700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc95843560 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fde00001eb0 tx=0x7fde00005fd0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 -- 172.21.15.38:0/1976632700 shutdown_connections 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 --2- 172.21.15.38:0/1976632700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc9583ec40 0x55fc95843560 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 --2- 172.21.15.38:0/1976632700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc95839ad0 0x55fc958584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 --2- 172.21.15.38:0/1976632700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc958390e0 0x55fc95842140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 -- 172.21.15.38:0/1976632700 >> 172.21.15.38:0/1976632700 conn(0x55fc95801400 msgr2=0x55fc9584f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 -- 172.21.15.38:0/1976632700 shutdown_connections 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 -- 172.21.15.38:0/1976632700 wait complete. 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 -- start start 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 Processor -- start 2024-03-26T06:37:18.325 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc958390e0 0x55fc958a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc958a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc9583ec40 0x55fc958a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc958511c0 con 0x55fc95839ad0 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc95850b60 con 0x55fc9583ec40 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc95851820 con 0x55fc958390e0 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1fb76740 10 client.0 _ll_register_callbacks cb 0x7fde0800b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc958a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1659c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc958a2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42236/0 (socket says 172.21.15.38:42236) 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde1659c640 1 -- 172.21.15.38:0/2983140700 learned_addr learned my addr 172.21.15.38:0/2983140700 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde0ffff640 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc958390e0 0x55fc958a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.317+0000 7fde0ffff640 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc9583ec40 0x55fc958a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.326 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1659c640 1 -- 172.21.15.38:0/2983140700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc958390e0 msgr2=0x55fc958a29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1659c640 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc958390e0 0x55fc958a29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1659c640 1 -- 172.21.15.38:0/2983140700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc9583ec40 msgr2=0x55fc958a9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1659c640 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc9583ec40 0x55fc958a9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1659c640 1 -- 172.21.15.38:0/2983140700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc958aa550 con 0x55fc95839ad0 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1659c640 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc958a2f20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fde08019040 tx=0x7fde0800b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0450b6588f8754 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fddfdffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fddfdffb640 1 -- 172.21.15.38:0/2983140700 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0800f4a0 con 0x55fc95839ad0 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fddfdffb640 1 -- 172.21.15.38:0/2983140700 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fde0800f640 con 0x55fc95839ad0 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1fb76740 1 -- 172.21.15.38:0/2983140700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fde00004470 con 0x55fc95839ad0 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fddfdffb640 1 -- 172.21.15.38:0/2983140700 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde0800f840 con 0x55fc95839ad0 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1fb76740 10 client.15222 fetch_fsmap learned FSMap version 22 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1fb76740 1 -- 172.21.15.38:0/2983140700 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55fc958ab880 con 0x55fc95839ad0 2024-03-26T06:37:18.327 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fddfdffb640 1 -- 172.21.15.38:0/2983140700 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde00004470 con 0x55fc95839ad0 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fddfdffb640 1 -- 172.21.15.38:0/2983140700 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fde0802b070 con 0x55fc95839ad0 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1fb76740 10 client.15222 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1fb76740 10 client.15222 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1fb76740 20 client.15222 populate_metadata read hostname 'smithi038' 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x55fc958afb80 0x55fc958abab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1fb76740 4 client.15222 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.318+0000 7fde1fb76740 1 -- 172.21.15.38:0/2983140700 --> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55fc957a6050 con 0x55fc958afb80 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde0ffff640 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x55fc958afb80 0x55fc958abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde0ffff640 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x55fc958afb80 0x55fc958abab0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ffb4b91664f7dc6e server_cookie=13ceadaefc9f178e in_seq=0 out_seq=0 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fddfdffb640 10 client.15222 ms_handle_connect on v2:172.21.15.121:6836/1114423776 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fddfdffb640 1 -- 172.21.15.38:0/2983140700 <== mds.2 v2:172.21.15.121:6836/1114423776 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x55fc957a6050 con 0x55fc958afb80 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fddfdffb640 10 client.15222 handle_command_reply: tid=0 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde1fb76740 1 client.15222 shutdown 2024-03-26T06:37:18.328 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde1fb76740 1 -- 172.21.15.38:0/2983140700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 msgr2=0x55fc958a2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc958a2f20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fde08019040 tx=0x7fde0800b9a0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde1fb76740 1 -- 172.21.15.38:0/2983140700 shutdown_connections 2024-03-26T06:37:18.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.121:6836/1114423776,v1:172.21.15.121:6837/1114423776] conn(0x55fc958afb80 0x55fc958abab0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc9583ec40 0x55fc958a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc95839ad0 0x55fc958a2f20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde1fb76740 1 --2- 172.21.15.38:0/2983140700 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc958390e0 0x55fc958a29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fddfdffb640 0 client.15222 ms_handle_reset on v2:172.21.15.121:6836/1114423776 2024-03-26T06:37:18.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde1fb76740 1 -- 172.21.15.38:0/2983140700 >> 172.21.15.38:0/2983140700 conn(0x55fc95801400 msgr2=0x55fc958584c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde1fb76740 1 -- 172.21.15.38:0/2983140700 shutdown_connections 2024-03-26T06:37:18.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.319+0000 7fde1fb76740 1 -- 172.21.15.38:0/2983140700 wait complete. 2024-03-26T06:37:18.329 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1fb76740 20 client.15222 trim_cache size 0 max 16384 2024-03-26T06:37:18.330 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:37:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c2824556-eb39-11ee-95d0-87774f69a715", 2024-03-26T06:37:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 2, 2024-03-26T06:37:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24488, 2024-03-26T06:37:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-03-26T06:37:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-03-26T06:37:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-03-26T06:37:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 107.15983502, 2024-03-26T06:37:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 22, 2024-03-26T06:37:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 88, 2024-03-26T06:37:18.330 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 86, 2024-03-26T06:37:18.331 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 127.69372502100001 2024-03-26T06:37:18.331 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:37:18.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 -- 172.21.15.38:0/2147123491 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fde040a2800 msgr2=0x7fde040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fde040a2800 0x7fde040a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fde1006e170 tx=0x7fde100c5ec0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.331 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 -- 172.21.15.38:0/2147123491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde18101dc0 msgr2=0x7fde1819e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde18101dc0 0x7fde1819e9a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fde1000cad0 tx=0x7fde1000cf90 comp rx=0 tx=0).stop 2024-03-26T06:37:18.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 -- 172.21.15.38:0/2147123491 shutdown_connections 2024-03-26T06:37:18.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fde040a2800 0x7fde040a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fde18101dc0 0x7fde1819e9a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fde18101400 0x7fde1819c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 --2- 172.21.15.38:0/2147123491 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde18100210 0x7fde1806a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 -- 172.21.15.38:0/2147123491 >> 172.21.15.38:0/2147123491 conn(0x7fde180fb7a0 msgr2=0x7fde18104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 -- 172.21.15.38:0/2147123491 shutdown_connections 2024-03-26T06:37:18.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.320+0000 7fde1cebf640 1 -- 172.21.15.38:0/2147123491 wait complete. 2024-03-26T06:37:18.333 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-26T06:37:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.434+0000 7f0069953640 1 -- 172.21.15.38:0/1070556722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0064106fe0 msgr2=0x7f0064109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.434+0000 7f0069953640 1 --2- 172.21.15.38:0/1070556722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0064106fe0 0x7f0064109430 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0054009ca0 tx=0x7f005402c0f0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.434+0000 7f0069953640 1 -- 172.21.15.38:0/1070556722 shutdown_connections 2024-03-26T06:37:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.434+0000 7f0069953640 1 --2- 172.21.15.38:0/1070556722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0064106fe0 0x7f0064109430 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.434+0000 7f0069953640 1 --2- 172.21.15.38:0/1070556722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00641046b0 0x7f0064106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.434+0000 7f0069953640 1 --2- 172.21.15.38:0/1070556722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0064101d80 0x7f0064104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.434+0000 7f0069953640 1 -- 172.21.15.38:0/1070556722 >> 172.21.15.38:0/1070556722 conn(0x7f00640fb790 msgr2=0x7f00640fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.434+0000 7f0069953640 1 -- 172.21.15.38:0/1070556722 shutdown_connections 2024-03-26T06:37:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.434+0000 7f0069953640 1 -- 172.21.15.38:0/1070556722 wait complete. 2024-03-26T06:37:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.435+0000 7f0069953640 1 Processor -- start 2024-03-26T06:37:18.439 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.435+0000 7f0069953640 1 -- start start 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.435+0000 7f0069953640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0064101d80 0x7f00641975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.435+0000 7f0069953640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00641046b0 0x7f0064197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.435+0000 7f0069953640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0064106fe0 0x7f006419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.435+0000 7f0069953640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006410cf90 con 0x7f0064106fe0 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.435+0000 7f0069953640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006410ce10 con 0x7f0064101d80 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.435+0000 7f0069953640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006410d110 con 0x7f00641046b0 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.435+0000 7f00637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0064106fe0 0x7f006419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.435+0000 7f00637fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0064106fe0 0x7f006419eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42244/0 (socket says 172.21.15.38:42244) 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.435+0000 7f00637fe640 1 -- 172.21.15.38:0/2592942992 learned_addr learned my addr 172.21.15.38:0/2592942992 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.436+0000 7f00627fc640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00641046b0 0x7f0064197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.436+0000 7f0062ffd640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0064101d80 0x7f00641975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.436+0000 7f00637fe640 1 -- 172.21.15.38:0/2592942992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00641046b0 msgr2=0x7f0064197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.440 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.436+0000 7f00637fe640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00641046b0 0x7f0064197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.436+0000 7f00637fe640 1 -- 172.21.15.38:0/2592942992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0064101d80 msgr2=0x7f00641975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.436+0000 7f00637fe640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0064101d80 0x7f00641975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.436+0000 7f00637fe640 1 -- 172.21.15.38:0/2592942992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006419f210 con 0x7f0064106fe0 2024-03-26T06:37:18.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.436+0000 7f0062ffd640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0064101d80 0x7f00641975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:37:18.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.436+0000 7f00637fe640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0064106fe0 0x7f006419eba0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0054037040 tx=0x7f0054032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.441 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.438+0000 7f0068951640 1 -- 172.21.15.38:0/2592942992 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0054039040 con 0x7f0064106fe0 2024-03-26T06:37:18.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.438+0000 7f0068951640 1 -- 172.21.15.38:0/2592942992 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0054044070 con 0x7f0064106fe0 2024-03-26T06:37:18.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.438+0000 7f0068951640 1 -- 172.21.15.38:0/2592942992 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0054004330 con 0x7f0064106fe0 2024-03-26T06:37:18.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.438+0000 7f0069953640 1 -- 172.21.15.38:0/2592942992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006419f4a0 con 0x7f0064106fe0 2024-03-26T06:37:18.442 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.438+0000 7f0069953640 1 -- 172.21.15.38:0/2592942992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006419f990 con 0x7f0064106fe0 2024-03-26T06:37:18.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.441+0000 7f0068951640 1 -- 172.21.15.38:0/2592942992 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0054003a00 con 0x7f0064106fe0 2024-03-26T06:37:18.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.441+0000 7f006c60d740 1 Processor -- start 2024-03-26T06:37:18.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.441+0000 7f006c60d740 1 -- start start 2024-03-26T06:37:18.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.441+0000 7f006c60d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc6a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.441+0000 7f006c60d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6aad0 0x564eecc6f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.441+0000 7f006c60d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6fc40 0x564eecc720d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.441+0000 7f006c60d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecbcf7c0 con 0x564eecc6a0e0 2024-03-26T06:37:18.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.441+0000 7f006c60d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecb78b50 con 0x564eecc6aad0 2024-03-26T06:37:18.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.441+0000 7f006c60d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecbcb860 con 0x564eecc6fc40 2024-03-26T06:37:18.445 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f0068951640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f004c0a2800 0x7f004c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00627fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc6a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00627fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc6a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42260/0 (socket says 172.21.15.38:42260) 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00627fc640 1 -- 172.21.15.38:0/3377160925 learned_addr learned my addr 172.21.15.38:0/3377160925 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f0062ffd640 1 --2- 172.21.15.38:0/3377160925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6aad0 0x564eecc6f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00637fe640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f004c0a2800 0x7f004c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00627fc640 1 --2- 172.21.15.38:0/3377160925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6fc40 0x564eecc720d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f0068951640 1 -- 172.21.15.38:0/2592942992 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f00540f1210 con 0x7f0064106fe0 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00627fc640 1 -- 172.21.15.38:0/3377160925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6fc40 msgr2=0x564eecc720d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00627fc640 1 --2- 172.21.15.38:0/3377160925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6fc40 0x564eecc720d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00627fc640 1 -- 172.21.15.38:0/3377160925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6aad0 msgr2=0x564eecc6f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00627fc640 1 --2- 172.21.15.38:0/3377160925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6aad0 0x564eecc6f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00627fc640 1 -- 172.21.15.38:0/3377160925 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564eecb8c9a0 con 0x564eecc6a0e0 2024-03-26T06:37:18.446 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00627fc640 1 --2- 172.21.15.38:0/3377160925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc6a500 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0058014990 tx=0x7f005800a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=234f90146bba894c server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.447 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.442+0000 7f00637fe640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f004c0a2800 0x7f004c0a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f00540ba580 tx=0x7f005408d6d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.444+0000 7f0049ffb640 1 -- 172.21.15.38:0/3377160925 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058004220 con 0x564eecc6a0e0 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.444+0000 7f0049ffb640 1 -- 172.21.15.38:0/3377160925 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00580043c0 con 0x564eecc6a0e0 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.444+0000 7f0049ffb640 1 -- 172.21.15.38:0/3377160925 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058004b80 con 0x564eecc6a0e0 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.445+0000 7f006c60d740 1 -- 172.21.15.38:0/3377160925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 msgr2=0x564eecc6a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.445+0000 7f006c60d740 1 --2- 172.21.15.38:0/3377160925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc6a500 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f0058014990 tx=0x7f005800a630 comp rx=0 tx=0).stop 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.445+0000 7f006c60d740 1 -- 172.21.15.38:0/3377160925 shutdown_connections 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.445+0000 7f006c60d740 1 --2- 172.21.15.38:0/3377160925 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6fc40 0x564eecc720d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.445+0000 7f006c60d740 1 --2- 172.21.15.38:0/3377160925 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6aad0 0x564eecc6f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.445+0000 7f006c60d740 1 --2- 172.21.15.38:0/3377160925 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc6a500 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.445+0000 7f006c60d740 1 -- 172.21.15.38:0/3377160925 >> 172.21.15.38:0/3377160925 conn(0x564eecc32400 msgr2=0x564eecc65a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.445+0000 7f006c60d740 1 -- 172.21.15.38:0/3377160925 shutdown_connections 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.445+0000 7f006c60d740 1 -- 172.21.15.38:0/3377160925 wait complete. 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.445+0000 7f006c60d740 1 -- start start 2024-03-26T06:37:18.452 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f006c60d740 1 Processor -- start 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f006c60d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f006c60d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 0x564eecc74ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f006c60d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6fc40 0x564eecc7bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f006c60d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecc72b90 con 0x564eecc6a0e0 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f006c60d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecb8bf90 con 0x564eecc6fc40 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f006c60d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecc72d10 con 0x564eecc6aad0 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f006c60d740 10 client.0 _ll_register_callbacks cb 0x564eecbc7400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f00627fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f00627fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42272/0 (socket says 172.21.15.38:42272) 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f00627fc640 1 -- 172.21.15.38:0/1753807289 learned_addr learned my addr 172.21.15.38:0/1753807289 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f00627fc640 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6fc40 0x564eecc7bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f0062ffd640 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 0x564eecc74ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f00627fc640 1 -- 172.21.15.38:0/1753807289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 msgr2=0x564eecc74ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f00627fc640 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 0x564eecc74ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.453 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f00627fc640 1 -- 172.21.15.38:0/1753807289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6fc40 msgr2=0x564eecc7bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f00627fc640 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6fc40 0x564eecc7bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f00627fc640 1 -- 172.21.15.38:0/1753807289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564eec8e42d0 con 0x564eecc6a0e0 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f0062ffd640 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 0x564eecc74ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f00627fc640 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74590 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f005800adb0 tx=0x7f005800ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13be01cdeffdf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.446+0000 7f0042ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f0042ffd640 1 -- 172.21.15.38:0/1753807289 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058028020 con 0x564eecc6a0e0 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f0042ffd640 1 -- 172.21.15.38:0/1753807289 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f005800db50 con 0x564eecc6a0e0 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f006c60d740 1 -- 172.21.15.38:0/1753807289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f00580043c0 con 0x564eecc6a0e0 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f0042ffd640 1 -- 172.21.15.38:0/1753807289 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f005800dd50 con 0x564eecc6a0e0 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f006c60d740 10 client.15234 fetch_fsmap learned FSMap version 22 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f006c60d740 1 -- 172.21.15.38:0/1753807289 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x564eec9ae6b0 con 0x564eecc6a0e0 2024-03-26T06:37:18.454 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f0042ffd640 1 -- 172.21.15.38:0/1753807289 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00580043c0 con 0x564eecc6a0e0 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f0042ffd640 1 -- 172.21.15.38:0/1753807289 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0058012450 con 0x564eecc6a0e0 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f006c60d740 10 client.15234 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f006c60d740 10 client.15234 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f006c60d740 20 client.15234 populate_metadata read hostname 'smithi038' 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f006c60d740 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x564eecc80c50 0x564eecc850d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f006c60d740 4 client.15234 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.447+0000 7f006c60d740 1 -- 172.21.15.38:0/1753807289 --> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564eecbdcd00 con 0x564eecc80c50 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.448+0000 7f0062ffd640 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x564eecc80c50 0x564eecc850d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.448+0000 7f0042ffd640 10 client.15234 ms_handle_connect on v2:172.21.15.170:6838/2204378068 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.448+0000 7f0062ffd640 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x564eecc80c50 0x564eecc850d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2ab5e7f46e47d949 server_cookie=d1f6555dbee72f08 in_seq=0 out_seq=0 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f0042ffd640 1 -- 172.21.15.38:0/1753807289 <== mds.1 v2:172.21.15.170:6838/2204378068 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564eecbdcd00 con 0x564eecc80c50 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f0042ffd640 10 client.15234 handle_command_reply: tid=0 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 1 client.15234 shutdown 2024-03-26T06:37:18.455 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 1 -- 172.21.15.38:0/1753807289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 msgr2=0x564eecc74590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74590 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f005800adb0 tx=0x7f005800ac00 comp rx=0 tx=0).stop 2024-03-26T06:37:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 1 -- 172.21.15.38:0/1753807289 shutdown_connections 2024-03-26T06:37:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x564eecc80c50 0x564eecc850d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6fc40 0x564eecc7bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 0x564eecc74ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f0042ffd640 0 client.15234 ms_handle_reset on v2:172.21.15.170:6838/2204378068 2024-03-26T06:37:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 1 --2- 172.21.15.38:0/1753807289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74590 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 1 -- 172.21.15.38:0/1753807289 >> 172.21.15.38:0/1753807289 conn(0x564eecc32400 msgr2=0x564eecc64c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 1 -- 172.21.15.38:0/1753807289 shutdown_connections 2024-03-26T06:37:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 1 -- 172.21.15.38:0/1753807289 wait complete. 2024-03-26T06:37:18.456 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.449+0000 7f006c60d740 20 client.15234 trim_cache size 0 max 16384 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f006c60d740 1 Processor -- start 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f006c60d740 1 -- start start 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f006c60d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f006c60d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 0x564eecc7baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f006c60d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6fc40 0x564eecc89800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f006c60d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecc80470 con 0x564eecc6a0e0 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f006c60d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecc80ad0 con 0x564eecc6fc40 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f006c60d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecc81130 con 0x564eecc6aad0 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f00627fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f00627fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42280/0 (socket says 172.21.15.38:42280) 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f00627fc640 1 -- 172.21.15.38:0/2808226953 learned_addr learned my addr 172.21.15.38:0/2808226953 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f0062ffd640 1 --2- 172.21.15.38:0/2808226953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 0x564eecc7baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f00627fc640 1 -- 172.21.15.38:0/2808226953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 msgr2=0x564eecc7baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f00627fc640 1 --2- 172.21.15.38:0/2808226953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 0x564eecc7baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f00627fc640 1 -- 172.21.15.38:0/2808226953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6fc40 msgr2=0x564eecc89800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f00627fc640 1 --2- 172.21.15.38:0/2808226953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6fc40 0x564eecc89800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f00627fc640 1 -- 172.21.15.38:0/2808226953 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564eecc75580 con 0x564eecc6a0e0 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f0062ffd640 1 --2- 172.21.15.38:0/2808226953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 0x564eecc7baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f00627fc640 1 --2- 172.21.15.38:0/2808226953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74a30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0058000c00 tx=0x7f005800def0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b30ef5066d6f92 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.465+0000 7f00497fa640 1 -- 172.21.15.38:0/2808226953 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058028050 con 0x564eecc6a0e0 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f00497fa640 1 -- 172.21.15.38:0/2808226953 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0058012a10 con 0x564eecc6a0e0 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f00497fa640 1 -- 172.21.15.38:0/2808226953 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058005320 con 0x564eecc6a0e0 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f006c60d740 1 -- 172.21.15.38:0/2808226953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 msgr2=0x564eecc74a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f006c60d740 1 --2- 172.21.15.38:0/2808226953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74a30 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0058000c00 tx=0x7f005800def0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f006c60d740 1 -- 172.21.15.38:0/2808226953 shutdown_connections 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f006c60d740 1 --2- 172.21.15.38:0/2808226953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6fc40 0x564eecc89800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f006c60d740 1 --2- 172.21.15.38:0/2808226953 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6aad0 0x564eecc7baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f006c60d740 1 --2- 172.21.15.38:0/2808226953 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6a0e0 0x564eecc74a30 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f006c60d740 1 -- 172.21.15.38:0/2808226953 >> 172.21.15.38:0/2808226953 conn(0x564eecc32400 msgr2=0x564eecc890f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f006c60d740 1 -- 172.21.15.38:0/2808226953 shutdown_connections 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f006c60d740 1 -- 172.21.15.38:0/2808226953 wait complete. 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.466+0000 7f006c60d740 1 -- start start 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f006c60d740 1 Processor -- start 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f006c60d740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6a0e0 0x564eecc7bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f006c60d740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6aad0 0x564eecc785b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f006c60d740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6fc40 0x564eecc78bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f006c60d740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecc81df0 con 0x564eecc6aad0 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f006c60d740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecc81790 con 0x564eecc6a0e0 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f006c60d740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x564eecc82450 con 0x564eecc6fc40 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f006c60d740 10 client.0 _ll_register_callbacks cb 0x7f005800a9e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:37:18.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f0062ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6aad0 0x564eecc785b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f0062ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6aad0 0x564eecc785b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42292/0 (socket says 172.21.15.38:42292) 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f0062ffd640 1 -- 172.21.15.38:0/1279728766 learned_addr learned my addr 172.21.15.38:0/1279728766 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f00627fc640 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6a0e0 0x564eecc7bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f0062ffd640 1 -- 172.21.15.38:0/1279728766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6fc40 msgr2=0x564eecc78bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f0062ffd640 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6fc40 0x564eecc78bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f0062ffd640 1 -- 172.21.15.38:0/1279728766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6a0e0 msgr2=0x564eecc7bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f0062ffd640 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6a0e0 0x564eecc7bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.467+0000 7f0062ffd640 1 -- 172.21.15.38:0/1279728766 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564eeccd6e10 con 0x564eecc6aad0 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f0062ffd640 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6aad0 0x564eecc785b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f005c00cc70 tx=0x7f005c005410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8bf5b0a92ea1f5a server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f0042ffd640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f0042ffd640 1 -- 172.21.15.38:0/1279728766 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c004de0 con 0x564eecc6aad0 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f0042ffd640 1 -- 172.21.15.38:0/1279728766 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f005c019220 con 0x564eecc6aad0 2024-03-26T06:37:18.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f006c60d740 1 -- 172.21.15.38:0/1279728766 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0058012a10 con 0x564eecc6aad0 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f0042ffd640 1 -- 172.21.15.38:0/1279728766 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f005c019420 con 0x564eecc6aad0 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f006c60d740 10 client.15240 fetch_fsmap learned FSMap version 22 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f006c60d740 1 -- 172.21.15.38:0/1279728766 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x564eeccd8180 con 0x564eecc6aad0 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f0042ffd640 1 -- 172.21.15.38:0/1279728766 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0058012a10 con 0x564eecc6aad0 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f0042ffd640 1 -- 172.21.15.38:0/1279728766 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f005c011c80 con 0x564eecc6aad0 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f006c60d740 10 client.15240 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f006c60d740 10 client.15240 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f006c60d740 20 client.15240 populate_metadata read hostname 'smithi038' 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f006c60d740 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x564eeccdc7f0 0x564eeccd8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f006c60d740 4 client.15240 mds_command: new command op to 24502 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.468+0000 7f006c60d740 1 -- 172.21.15.38:0/1279728766 --> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564eecbd7050 con 0x564eeccdc7f0 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.469+0000 7f00627fc640 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x564eeccdc7f0 0x564eeccd8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.469+0000 7f00627fc640 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x564eeccdc7f0 0x564eeccd8470 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=20cf7cfaa8eef4dd server_cookie=8397eea7490cc3cc in_seq=0 out_seq=0 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.469+0000 7f0042ffd640 10 client.15240 ms_handle_connect on v2:172.21.15.170:6838/2204378068 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.469+0000 7f0042ffd640 1 -- 172.21.15.38:0/1279728766 <== mds.1 v2:172.21.15.170:6838/2204378068 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x564eecbd7050 con 0x564eeccdc7f0 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.469+0000 7f0042ffd640 10 client.15240 handle_command_reply: tid=0 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.469+0000 7f006c60d740 1 client.15240 shutdown 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.469+0000 7f006c60d740 1 -- 172.21.15.38:0/1279728766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6aad0 msgr2=0x564eecc785b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.469+0000 7f006c60d740 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6aad0 0x564eecc785b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f005c00cc70 tx=0x7f005c005410 comp rx=0 tx=0).stop 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.469+0000 7f006c60d740 1 -- 172.21.15.38:0/1279728766 shutdown_connections 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f006c60d740 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.170:6838/2204378068,v1:172.21.15.170:6839/2204378068] conn(0x564eeccdc7f0 0x564eeccd8470 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f006c60d740 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x564eecc6fc40 0x564eecc78bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f006c60d740 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564eecc6aad0 0x564eecc785b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0042ffd640 0 client.15240 ms_handle_reset on v2:172.21.15.170:6838/2204378068 2024-03-26T06:37:18.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f006c60d740 1 --2- 172.21.15.38:0/1279728766 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x564eecc6a0e0 0x564eecc7bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f006c60d740 1 -- 172.21.15.38:0/1279728766 >> 172.21.15.38:0/1279728766 conn(0x564eecc32400 msgr2=0x564eecc89980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f006c60d740 1 -- 172.21.15.38:0/1279728766 shutdown_connections 2024-03-26T06:37:18.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f006c60d740 1 -- 172.21.15.38:0/1279728766 wait complete. 2024-03-26T06:37:18.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f006c60d740 20 client.15240 trim_cache size 0 max 16384 2024-03-26T06:37:18.479 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:37:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c2824556-eb39-11ee-95d0-87774f69a715", 2024-03-26T06:37:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 1, 2024-03-26T06:37:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24502, 2024-03-26T06:37:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-03-26T06:37:18.479 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-03-26T06:37:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-03-26T06:37:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 108.321639467, 2024-03-26T06:37:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 22, 2024-03-26T06:37:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 88, 2024-03-26T06:37:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 86, 2024-03-26T06:37:18.480 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 125.13918086699999 2024-03-26T06:37:18.480 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:37:18.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 -- 172.21.15.38:0/2592942992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f004c0a2800 msgr2=0x7f004c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f004c0a2800 0x7f004c0a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f00540ba580 tx=0x7f005408d6d0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 -- 172.21.15.38:0/2592942992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0064106fe0 msgr2=0x7f006419eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0064106fe0 0x7f006419eba0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0054037040 tx=0x7f0054032f90 comp rx=0 tx=0).stop 2024-03-26T06:37:18.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 -- 172.21.15.38:0/2592942992 shutdown_connections 2024-03-26T06:37:18.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f004c0a2800 0x7f004c0a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0064106fe0 0x7f006419eba0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f00641046b0 0x7f0064197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 --2- 172.21.15.38:0/2592942992 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0064101d80 0x7f00641975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 -- 172.21.15.38:0/2592942992 >> 172.21.15.38:0/2592942992 conn(0x7f00640fb790 msgr2=0x7f00641026c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 -- 172.21.15.38:0/2592942992 shutdown_connections 2024-03-26T06:37:18.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.470+0000 7f0069953640 1 -- 172.21.15.38:0/2592942992 wait complete. 2024-03-26T06:37:18.483 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-26T06:37:18.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.584+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/404049958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9dd01001f0 msgr2=0x7f9dd0100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.584+0000 7f9dd4d1b640 1 --2- 172.21.15.38:0/404049958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9dd01001f0 0x7f9dd0100610 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009900 tx=0x7f9dc002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:18.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.585+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/404049958 shutdown_connections 2024-03-26T06:37:18.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.585+0000 7f9dd4d1b640 1 --2- 172.21.15.38:0/404049958 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd0101da0 0x7f9dd0108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.585+0000 7f9dd4d1b640 1 --2- 172.21.15.38:0/404049958 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dd01013e0 0x7f9dd0101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.585+0000 7f9dd4d1b640 1 --2- 172.21.15.38:0/404049958 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9dd01001f0 0x7f9dd0100610 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.585+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/404049958 >> 172.21.15.38:0/404049958 conn(0x7f9dd00fb780 msgr2=0x7f9dd00fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.585+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/404049958 shutdown_connections 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.585+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/404049958 wait complete. 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.585+0000 7f9dd4d1b640 1 Processor -- start 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.585+0000 7f9dd4d1b640 1 -- start start 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.585+0000 7f9dd4d1b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9dd01001f0 0x7f9dd01976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.585+0000 7f9dd4d1b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dd01013e0 0x7f9dd0197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dd4d1b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd0101da0 0x7f9dd019ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dd4d1b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd010bec0 con 0x7f9dd01001f0 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dd4d1b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd010bd40 con 0x7f9dd0101da0 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dd4d1b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dd010c040 con 0x7f9dd01013e0 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dce59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9dd01001f0 0x7f9dd01976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dce59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9dd01001f0 0x7f9dd01976e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42318/0 (socket says 172.21.15.38:42318) 2024-03-26T06:37:18.590 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dce59c640 1 -- 172.21.15.38:0/351860292 learned_addr learned my addr 172.21.15.38:0/351860292 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dc7fff640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dd01013e0 0x7f9dd0197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dce59c640 1 -- 172.21.15.38:0/351860292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dd01013e0 msgr2=0x7f9dd0197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dce59c640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dd01013e0 0x7f9dd0197c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dce59c640 1 -- 172.21.15.38:0/351860292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd0101da0 msgr2=0x7f9dd019ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dce59c640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd0101da0 0x7f9dd019ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dce59c640 1 -- 172.21.15.38:0/351860292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd019f300 con 0x7f9dd01001f0 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dce59c640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9dd01001f0 0x7f9dd01976e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0005ab0 tx=0x7f9dc0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dc77fe640 1 -- 172.21.15.38:0/351860292 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0041070 con 0x7f9dd01001f0 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dc77fe640 1 -- 172.21.15.38:0/351860292 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dc00054e0 con 0x7f9dd01001f0 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/351860292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dd019f590 con 0x7f9dd01001f0 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dc77fe640 1 -- 172.21.15.38:0/351860292 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0034440 con 0x7f9dd01001f0 2024-03-26T06:37:18.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.586+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/351860292 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dd019fa80 con 0x7f9dd01001f0 2024-03-26T06:37:18.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.588+0000 7f9dc77fe640 1 -- 172.21.15.38:0/351860292 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9dc0044070 con 0x7f9dd01001f0 2024-03-26T06:37:18.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dd79d1740 1 Processor -- start 2024-03-26T06:37:18.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dd79d1740 1 -- start start 2024-03-26T06:37:18.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e74910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e799e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9e7c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9dd57c0 con 0x55e9c9e74ee0 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9d7eb50 con 0x55e9c9e744f0 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9dd1860 con 0x55e9c9e7a050 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dce59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e799e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dc77fe640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9dbc0a2800 0x7f9dbc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dce59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e799e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42334/0 (socket says 172.21.15.38:42334) 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dce59c640 1 -- 172.21.15.38:0/2367966138 learned_addr learned my addr 172.21.15.38:0/2367966138 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dced9d640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9dbc0a2800 0x7f9dbc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.589+0000 7f9dced9d640 1 --2- 172.21.15.38:0/2367966138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e74910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dc7fff640 1 --2- 172.21.15.38:0/2367966138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9e7c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dc77fe640 1 -- 172.21.15.38:0/351860292 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9dc00f0780 con 0x7f9dd01001f0 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dce59c640 1 -- 172.21.15.38:0/2367966138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 msgr2=0x55e9c9e7c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dce59c640 1 --2- 172.21.15.38:0/2367966138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9e7c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dce59c640 1 -- 172.21.15.38:0/2367966138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 msgr2=0x55e9c9e74910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dce59c640 1 --2- 172.21.15.38:0/2367966138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e74910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dce59c640 1 -- 172.21.15.38:0/2367966138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9c9d929a0 con 0x55e9c9e74ee0 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dce59c640 1 --2- 172.21.15.38:0/2367966138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e799e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00ee280 tx=0x7f9dc0009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c2cb74386d2e37 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dced9d640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9dbc0a2800 0x7f9dbc0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x55e9c9e75740 tx=0x7f9dc800b250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dc4ff9640 1 -- 172.21.15.38:0/2367966138 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc004b070 con 0x55e9c9e74ee0 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dc4ff9640 1 -- 172.21.15.38:0/2367966138 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dc00dd0c0 con 0x55e9c9e74ee0 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.590+0000 7f9dc4ff9640 1 -- 172.21.15.38:0/2367966138 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc00b9790 con 0x55e9c9e74ee0 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2367966138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 msgr2=0x55e9c9e799e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2367966138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e799e0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9dc00ee280 tx=0x7f9dc0009560 comp rx=0 tx=0).stop 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2367966138 shutdown_connections 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2367966138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9e7c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2367966138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e799e0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2367966138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e74910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2367966138 >> 172.21.15.38:0/2367966138 conn(0x55e9c9e38400 msgr2=0x55e9c9e6ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2367966138 shutdown_connections 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2367966138 wait complete. 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 -- start start 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 Processor -- start 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e7e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e7ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9e85d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9e7cdc0 con 0x55e9c9e74ee0 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9d91f90 con 0x55e9c9e744f0 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9e7cf40 con 0x55e9c9e7a050 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dd79d1740 10 client.0 _ll_register_callbacks cb 0x55e9c9dcd400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e7ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.591+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e7ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42338/0 (socket says 172.21.15.38:42338) 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dc7fff640 1 -- 172.21.15.38:0/2923114501 learned_addr learned my addr 172.21.15.38:0/2923114501 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dced9d640 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9e85d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dce59c640 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e7e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dc7fff640 1 -- 172.21.15.38:0/2923114501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 msgr2=0x55e9c9e85d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dc7fff640 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9e85d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dc7fff640 1 -- 172.21.15.38:0/2923114501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 msgr2=0x55e9c9e7e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dc7fff640 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e7e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dc7fff640 1 -- 172.21.15.38:0/2923114501 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e9c9aea2d0 con 0x55e9c9e74ee0 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dc7fff640 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e7ecc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9db800dd90 tx=0x7f9db800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460b0546e2e3e852 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9db5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9db5ffb640 1 -- 172.21.15.38:0/2923114501 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8011640 con 0x55e9c9e74ee0 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9db5ffb640 1 -- 172.21.15.38:0/2923114501 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9db80117e0 con 0x55e9c9e74ee0 2024-03-26T06:37:18.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2923114501 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e9c9de2d00 con 0x55e9c9e74ee0 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9db5ffb640 1 -- 172.21.15.38:0/2923114501 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9db80119e0 con 0x55e9c9e74ee0 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dd79d1740 10 client.15252 fetch_fsmap learned FSMap version 22 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2923114501 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e9c9bb46b0 con 0x55e9c9e74ee0 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.592+0000 7f9db5ffb640 1 -- 172.21.15.38:0/2923114501 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e9c9de2d00 con 0x55e9c9e74ee0 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.593+0000 7f9db5ffb640 1 -- 172.21.15.38:0/2923114501 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9db8019c00 con 0x55e9c9e74ee0 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.593+0000 7f9dd79d1740 10 client.15252 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.593+0000 7f9dd79d1740 10 client.15252 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.593+0000 7f9dd79d1740 20 client.15252 populate_metadata read hostname 'smithi038' 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.593+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55e9c9e8b3d0 0x55e9c9e8a430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.593+0000 7f9dd79d1740 4 client.15252 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.593+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2923114501 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e9c9ddd050 con 0x55e9c9e8b3d0 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.593+0000 7f9dce59c640 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55e9c9e8b3d0 0x55e9c9e8a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.593+0000 7f9dce59c640 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55e9c9e8b3d0 0x55e9c9e8a430 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dde7ffa7dcf94cb server_cookie=b614c8cfc4c74416 in_seq=0 out_seq=0 2024-03-26T06:37:18.603 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.593+0000 7f9db5ffb640 10 client.15252 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9db5ffb640 1 -- 172.21.15.38:0/2923114501 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e9c9ddd050 con 0x55e9c9e8b3d0 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9db5ffb640 10 client.15252 handle_command_reply: tid=0 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 1 client.15252 shutdown 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2923114501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 msgr2=0x55e9c9e7ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e7ecc0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9db800dd90 tx=0x7f9db800f730 comp rx=0 tx=0).stop 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2923114501 shutdown_connections 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55e9c9e8b3d0 0x55e9c9e8a430 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9e85d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e7ecc0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9db5ffb640 0 client.15252 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2923114501 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e7e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2923114501 >> 172.21.15.38:0/2923114501 conn(0x55e9c9e38400 msgr2=0x55e9c9e6ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2923114501 shutdown_connections 2024-03-26T06:37:18.604 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2923114501 wait complete. 2024-03-26T06:37:18.605 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.594+0000 7f9dd79d1740 20 client.15252 trim_cache size 0 max 16384 2024-03-26T06:37:18.617 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dd79d1740 1 Processor -- start 2024-03-26T06:37:18.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dd79d1740 1 -- start start 2024-03-26T06:37:18.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e7d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e85cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e7a050 0x55e9c9e93870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9e8abf0 con 0x55e9c9e7a050 2024-03-26T06:37:18.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9e8b250 con 0x55e9c9e744f0 2024-03-26T06:37:18.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9e8b8b0 con 0x55e9c9e74ee0 2024-03-26T06:37:18.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dced9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e7a050 0x55e9c9e93870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dced9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e7a050 0x55e9c9e93870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42344/0 (socket says 172.21.15.38:42344) 2024-03-26T06:37:18.618 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dced9d640 1 -- 172.21.15.38:0/2308724589 learned_addr learned my addr 172.21.15.38:0/2308724589 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dced9d640 1 -- 172.21.15.38:0/2308724589 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e74ee0 msgr2=0x55e9c9e85cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dced9d640 1 --2- 172.21.15.38:0/2308724589 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e85cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dced9d640 1 -- 172.21.15.38:0/2308724589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 msgr2=0x55e9c9e7d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dced9d640 1 --2- 172.21.15.38:0/2308724589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e7d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dced9d640 1 -- 172.21.15.38:0/2308724589 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e9c9e75770 con 0x55e9c9e7a050 2024-03-26T06:37:18.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.610+0000 7f9dced9d640 1 --2- 172.21.15.38:0/2308724589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e7a050 0x55e9c9e93870 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800b000 tx=0x7f9dc8006ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc231ad258e34d1d server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9db7fff640 1 -- 172.21.15.38:0/2308724589 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc80068a0 con 0x55e9c9e7a050 2024-03-26T06:37:18.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9db7fff640 1 -- 172.21.15.38:0/2308724589 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dc8003e50 con 0x55e9c9e7a050 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9db7fff640 1 -- 172.21.15.38:0/2308724589 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dc801f6a0 con 0x55e9c9e7a050 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2308724589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e7a050 msgr2=0x55e9c9e93870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2308724589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e7a050 0x55e9c9e93870 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f9dc800b000 tx=0x7f9dc8006ce0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2308724589 shutdown_connections 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2308724589 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e7a050 0x55e9c9e93870 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2308724589 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e85cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/2308724589 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e7d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2308724589 >> 172.21.15.38:0/2308724589 conn(0x55e9c9e38400 msgr2=0x55e9c9e75980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2308724589 shutdown_connections 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9dd79d1740 1 -- 172.21.15.38:0/2308724589 wait complete. 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.611+0000 7f9dd79d1740 1 -- start start 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dd79d1740 1 Processor -- start 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e828a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e82de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dd79d1740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9ee0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9e8c570 con 0x55e9c9e74ee0 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9e8bf10 con 0x55e9c9e744f0 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dd79d1740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e9c9e8cbd0 con 0x55e9c9e7a050 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dd79d1740 10 client.0 _ll_register_callbacks cb 0x7f9dc00bcef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e82de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dc7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e82de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42346/0 (socket says 172.21.15.38:42346) 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dc7fff640 1 -- 172.21.15.38:0/3573680593 learned_addr learned my addr 172.21.15.38:0/3573680593 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dced9d640 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9ee0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dc7fff640 1 -- 172.21.15.38:0/3573680593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 msgr2=0x55e9c9ee0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dc7fff640 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9ee0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dc7fff640 1 -- 172.21.15.38:0/3573680593 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 msgr2=0x55e9c9e828a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:18.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dc7fff640 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e828a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dc7fff640 1 -- 172.21.15.38:0/3573680593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e9c9e85e40 con 0x55e9c9e74ee0 2024-03-26T06:37:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dc7fff640 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e82de0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9db80184a0 tx=0x7f9db800d810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8250702ad4d59f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9db5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:37:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9db5ffb640 1 -- 172.21.15.38:0/3573680593 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db8030020 con 0x55e9c9e74ee0 2024-03-26T06:37:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9db5ffb640 1 -- 172.21.15.38:0/3573680593 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f9db8011d80 con 0x55e9c9e74ee0 2024-03-26T06:37:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.612+0000 7f9dd79d1740 1 -- 172.21.15.38:0/3573680593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9dc8003e50 con 0x55e9c9e74ee0 2024-03-26T06:37:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9db5ffb640 1 -- 172.21.15.38:0/3573680593 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9db8028040 con 0x55e9c9e74ee0 2024-03-26T06:37:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9dd79d1740 10 client.15258 fetch_fsmap learned FSMap version 22 2024-03-26T06:37:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9dd79d1740 1 -- 172.21.15.38:0/3573680593 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=4+}) v3 -- 0x55e9c9ee2400 con 0x55e9c9e74ee0 2024-03-26T06:37:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9db5ffb640 1 -- 172.21.15.38:0/3573680593 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9dc8003e50 con 0x55e9c9e74ee0 2024-03-26T06:37:18.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9db5ffb640 1 -- 172.21.15.38:0/3573680593 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 22) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9db80051d0 con 0x55e9c9e74ee0 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9dd79d1740 10 client.15258 fetch_fsmap finished waiting for FSMap version 22 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9dd79d1740 10 client.15258 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9dd79d1740 20 client.15258 populate_metadata read hostname 'smithi038' 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55e9c9ee6b30 0x55e9c9ee26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9dd79d1740 4 client.15258 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9dd79d1740 1 -- 172.21.15.38:0/3573680593 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e9c9d5b9a0 con 0x55e9c9ee6b30 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.613+0000 7f9dce59c640 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55e9c9ee6b30 0x55e9c9ee26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9db5ffb640 10 client.15258 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dce59c640 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55e9c9ee6b30 0x55e9c9ee26e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=440535e154150268 server_cookie=67ad89e8c5bd894c in_seq=0 out_seq=0 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9db5ffb640 1 -- 172.21.15.38:0/3573680593 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55e9c9d5b9a0 con 0x55e9c9ee6b30 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9db5ffb640 10 client.15258 handle_command_reply: tid=0 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 1 client.15258 shutdown 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 1 -- 172.21.15.38:0/3573680593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 msgr2=0x55e9c9e82de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e82de0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9db80184a0 tx=0x7f9db800d810 comp rx=0 tx=0).stop 2024-03-26T06:37:18.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 1 -- 172.21.15.38:0/3573680593 shutdown_connections 2024-03-26T06:37:18.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55e9c9ee6b30 0x55e9c9ee26e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55e9c9e7a050 0x55e9c9ee0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e9c9e74ee0 0x55e9c9e82de0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9db5ffb640 0 client.15258 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:37:18.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 1 --2- 172.21.15.38:0/3573680593 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55e9c9e744f0 0x55e9c9e828a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 1 -- 172.21.15.38:0/3573680593 >> 172.21.15.38:0/3573680593 conn(0x55e9c9e38400 msgr2=0x55e9c9e7d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 1 -- 172.21.15.38:0/3573680593 shutdown_connections 2024-03-26T06:37:18.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 1 -- 172.21.15.38:0/3573680593 wait complete. 2024-03-26T06:37:18.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd79d1740 20 client.15258 trim_cache size 0 max 16384 2024-03-26T06:37:18.624 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c2824556-eb39-11ee-95d0-87774f69a715", 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 0, 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24539, 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 111.305881105, 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 22, 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 88, 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 86, 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 116.743625814 2024-03-26T06:37:18.625 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:37:18.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/351860292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9dbc0a2800 msgr2=0x7f9dbc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd4d1b640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9dbc0a2800 0x7f9dbc0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x55e9c9e75740 tx=0x7f9dc800b250 comp rx=0 tx=0).stop 2024-03-26T06:37:18.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/351860292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9dd01001f0 msgr2=0x7f9dd01976e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:18.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd4d1b640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9dd01001f0 0x7f9dd01976e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0005ab0 tx=0x7f9dc0005dc0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.614+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/351860292 shutdown_connections 2024-03-26T06:37:18.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.615+0000 7f9dd4d1b640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f9dbc0a2800 0x7f9dbc0a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.615+0000 7f9dd4d1b640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9dd0101da0 0x7f9dd019ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.615+0000 7f9dd4d1b640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9dd01013e0 0x7f9dd0197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.615+0000 7f9dd4d1b640 1 --2- 172.21.15.38:0/351860292 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9dd01001f0 0x7f9dd01976e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:18.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.615+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/351860292 >> 172.21.15.38:0/351860292 conn(0x7f9dd00fb780 msgr2=0x7f9dd00fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:18.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.615+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/351860292 shutdown_connections 2024-03-26T06:37:18.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:18.615+0000 7f9dd4d1b640 1 -- 172.21.15.38:0/351860292 wait complete. 2024-03-26T06:37:18.628 INFO:teuthology.run_tasks:Running task kclient... 2024-03-26T06:37:18.652 INFO:tasks.kclient:Mounting kernel clients... 2024-03-26T06:37:18.653 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-26T06:37:18.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:18.653 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-03-26T06:37:18.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:18.668 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link delete ceph-brx 2024-03-26T06:37:18.731 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:18 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1021284481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:37:18.731 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:18 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3080728784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:37:18.731 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:18 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1888587771' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:37:18.737 INFO:teuthology.orchestra.run.smithi038.stderr:Cannot find device "ceph-brx" 2024-03-26T06:37:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:37:18.739 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-26T06:37:18.739 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-26T06:37:18.739 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-26T06:37:18.739 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph fs ls 2024-03-26T06:37:18.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:18 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1021284481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:37:18.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:18 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3080728784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:37:18.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:18 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1888587771' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:37:18.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:18 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1021284481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:37:18.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:18 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3080728784' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:37:18.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:18 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1888587771' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:37:19.020 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:37:19.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:19 smithi038 ceph-mon[32265]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:37:19.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:19 smithi121 ceph-mon[36349]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:37:19.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:19 smithi170 ceph-mon[36912]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:37:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.820+0000 7fe0b4263700 1 -- 172.21.15.38:0/2807620269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0ac100fb0 msgr2=0x7fe0ac101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.820+0000 7fe0b4263700 1 --2- 172.21.15.38:0/2807620269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0ac100fb0 0x7fe0ac101390 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fe0a80099c0 tx=0x7fe0a8009cd0 comp rx=0 tx=0).stop 2024-03-26T06:37:20.835 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.821+0000 7fe0b4263700 1 -- 172.21.15.38:0/2807620269 shutdown_connections 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.821+0000 7fe0b4263700 1 --2- 172.21.15.38:0/2807620269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0ac106090 0x7fe0ac10dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.821+0000 7fe0b4263700 1 --2- 172.21.15.38:0/2807620269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0ac100fb0 0x7fe0ac101390 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.821+0000 7fe0b4263700 1 --2- 172.21.15.38:0/2807620269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ac101960 0x7fe0ac1059b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.821+0000 7fe0b4263700 1 -- 172.21.15.38:0/2807620269 >> 172.21.15.38:0/2807620269 conn(0x7fe0ac077e10 msgr2=0x7fe0ac078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.821+0000 7fe0b4263700 1 -- 172.21.15.38:0/2807620269 shutdown_connections 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.821+0000 7fe0b4263700 1 -- 172.21.15.38:0/2807620269 wait complete. 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.821+0000 7fe0b4263700 1 Processor -- start 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.821+0000 7fe0b4263700 1 -- start start 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b4263700 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ac100fb0 0x7fe0ac1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b4263700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0ac101960 0x7fe0ac198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b4263700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0ac106090 0x7fe0ac19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b4263700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac111170 con 0x7fe0ac106090 2024-03-26T06:37:20.836 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b4263700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac110ff0 con 0x7fe0ac101960 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b4263700 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac1112f0 con 0x7fe0ac100fb0 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b2800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0ac106090 0x7fe0ac19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b2800700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0ac106090 0x7fe0ac19cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42370/0 (socket says 172.21.15.38:42370) 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b2800700 1 -- 172.21.15.38:0/1486563119 learned_addr learned my addr 172.21.15.38:0/1486563119 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b1fff700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ac100fb0 0x7fe0ac1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b17fe700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0ac101960 0x7fe0ac198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b2800700 1 -- 172.21.15.38:0/1486563119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ac100fb0 msgr2=0x7fe0ac1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b2800700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ac100fb0 0x7fe0ac1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b2800700 1 -- 172.21.15.38:0/1486563119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0ac101960 msgr2=0x7fe0ac198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b2800700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0ac101960 0x7fe0ac198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b2800700 1 -- 172.21.15.38:0/1486563119 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac19d6e0 con 0x7fe0ac106090 2024-03-26T06:37:20.837 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b1fff700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ac100fb0 0x7fe0ac1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b17fe700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0ac101960 0x7fe0ac198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b2800700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0ac106090 0x7fe0ac19cfa0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400b970 tx=0x7fe0a400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0a2ffd700 1 -- 172.21.15.38:0/1486563119 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4019070 con 0x7fe0ac106090 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0a2ffd700 1 -- 172.21.15.38:0/1486563119 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0a4014440 con 0x7fe0ac106090 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b4263700 1 -- 172.21.15.38:0/1486563119 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac19d9d0 con 0x7fe0ac106090 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0a2ffd700 1 -- 172.21.15.38:0/1486563119 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a40075d0 con 0x7fe0ac106090 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.822+0000 7fe0b4263700 1 -- 172.21.15.38:0/1486563119 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac1a5250 con 0x7fe0ac106090 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.823+0000 7fe0b4263700 1 -- 172.21.15.38:0/1486563119 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ac068aa0 con 0x7fe0ac106090 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.824+0000 7fe0a2ffd700 1 -- 172.21.15.38:0/1486563119 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe0a401c070 con 0x7fe0ac106090 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.825+0000 7fe0a2ffd700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0980a2910 0x7fe0980a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.825+0000 7fe0a2ffd700 1 -- 172.21.15.38:0/1486563119 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fe0a40ce2f0 con 0x7fe0ac106090 2024-03-26T06:37:20.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.825+0000 7fe0b1fff700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0980a2910 0x7fe0980a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:20.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.825+0000 7fe0b1fff700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0980a2910 0x7fe0980a4dd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800b580 tx=0x7fe0a8009f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:20.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:20.829+0000 7fe0a2ffd700 1 -- 172.21.15.38:0/1486563119 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe0a409b390 con 0x7fe0ac106090 2024-03-26T06:37:21.080 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.078+0000 7fe0b4263700 1 -- 172.21.15.38:0/1486563119 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe0ac02d830 con 0x7fe0ac106090 2024-03-26T06:37:21.081 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.078+0000 7fe0a2ffd700 1 -- 172.21.15.38:0/1486563119 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe0a409ac90 con 0x7fe0ac106090 2024-03-26T06:37:21.081 INFO:teuthology.orchestra.run.smithi038.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-26T06:37:21.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.080+0000 7fe0b4263700 1 -- 172.21.15.38:0/1486563119 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0980a2910 msgr2=0x7fe0980a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:21.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.080+0000 7fe0b4263700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0980a2910 0x7fe0980a4dd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800b580 tx=0x7fe0a8009f30 comp rx=0 tx=0).stop 2024-03-26T06:37:21.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.080+0000 7fe0b4263700 1 -- 172.21.15.38:0/1486563119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0ac106090 msgr2=0x7fe0ac19cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:21.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.080+0000 7fe0b4263700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0ac106090 0x7fe0ac19cfa0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400b970 tx=0x7fe0a400bd30 comp rx=0 tx=0).stop 2024-03-26T06:37:21.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.080+0000 7fe0b4263700 1 -- 172.21.15.38:0/1486563119 shutdown_connections 2024-03-26T06:37:21.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.080+0000 7fe0b4263700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0ac106090 0x7fe0ac19cfa0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:21.083 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.080+0000 7fe0b4263700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe0ac100fb0 0x7fe0ac1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:21.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.080+0000 7fe0b4263700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fe0980a2910 0x7fe0980a4dd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:21.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.080+0000 7fe0b4263700 1 --2- 172.21.15.38:0/1486563119 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe0ac101960 0x7fe0ac198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:21.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.080+0000 7fe0b4263700 1 -- 172.21.15.38:0/1486563119 >> 172.21.15.38:0/1486563119 conn(0x7fe0ac077e10 msgr2=0x7fe0ac104f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:21.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.080+0000 7fe0b4263700 1 -- 172.21.15.38:0/1486563119 shutdown_connections 2024-03-26T06:37:21.084 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:21.081+0000 7fe0b4263700 1 -- 172.21.15.38:0/1486563119 wait complete. 2024-03-26T06:37:21.805 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:21 smithi038 ceph-mon[32265]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:37:21.806 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:21 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1486563119' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-26T06:37:21.833 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-26T06:37:21.833 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-26T06:37:21.833 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi038.front.sepia.ceph.com 2024-03-26T06:37:21.833 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-26T06:37:21.833 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-26T06:37:21.833 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-26T06:37:21.833 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-26T06:37:21.833 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-26T06:37:21.833 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-26T06:37:21.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:21.834 DEBUG:teuthology.orchestra.run.smithi038:> ip addr 2024-03-26T06:37:21.850 INFO:teuthology.orchestra.run.smithi038.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-26T06:37:21.850 INFO:teuthology.orchestra.run.smithi038.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-26T06:37:21.850 INFO:teuthology.orchestra.run.smithi038.stdout: inet 127.0.0.1/8 scope host lo 2024-03-26T06:37:21.850 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-03-26T06:37:21.850 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 ::1/128 scope host 2024-03-26T06:37:21.850 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-03-26T06:37:21.850 INFO:teuthology.orchestra.run.smithi038.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-26T06:37:21.850 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a2 brd ff:ff:ff:ff:ff:ff 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f0 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a3 brd ff:ff:ff:ff:ff:ff 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp7s0f1 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fc brd ff:ff:ff:ff:ff:ff 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f0 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout: inet 172.21.15.38/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft 41935sec preferred_lft 41935sec 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 fe80::ec4:7aff:febd:15fc/64 scope link noprefixroute 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-26T06:37:21.851 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fd brd ff:ff:ff:ff:ff:ff 2024-03-26T06:37:21.852 INFO:teuthology.orchestra.run.smithi038.stdout: altname enp1s0f1 2024-03-26T06:37:21.852 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-26T06:37:21.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:21.852 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2024-03-26T06:37:21.852 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-03-26T06:37:21.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add name ceph-brx type bridge 2024-03-26T06:37:21.852 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr flush dev ceph-brx 2024-03-26T06:37:21.853 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set ceph-brx up 2024-03-26T06:37:21.853 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-26T06:37:21.853 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-03-26T06:37:21.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:21 smithi121 ceph-mon[36349]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:37:21.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:21 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1486563119' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-26T06:37:21.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:21 smithi170 ceph-mon[36912]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-26T06:37:21.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:21 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1486563119' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-26T06:37:22.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:22.015 DEBUG:teuthology.orchestra.run.smithi038:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-26T06:37:22.042 INFO:teuthology.orchestra.run.smithi038.stdout:1 2024-03-26T06:37:22.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:22.043 DEBUG:teuthology.orchestra.run.smithi038:> route 2024-03-26T06:37:22.124 INFO:teuthology.orchestra.run.smithi038.stdout:Kernel IP routing table 2024-03-26T06:37:22.124 INFO:teuthology.orchestra.run.smithi038.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-26T06:37:22.124 INFO:teuthology.orchestra.run.smithi038.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-26T06:37:22.124 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-26T06:37:22.124 INFO:teuthology.orchestra.run.smithi038.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-26T06:37:22.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:22.124 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2024-03-26T06:37:22.125 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-03-26T06:37:22.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-26T06:37:22.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-26T06:37:22.125 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-26T06:37:22.125 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-03-26T06:37:22.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:22.379 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-03-26T06:37:22.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:22.394 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list-id 2024-03-26T06:37:22.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:22.449 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2024-03-26T06:37:22.449 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-03-26T06:37:22.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-26T06:37:22.449 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-26T06:37:22.449 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-03-26T06:37:22.533 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-26T06:37:22.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:22.533 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2024-03-26T06:37:22.533 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-03-26T06:37:22.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-26T06:37:22.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-26T06:37:22.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-26T06:37:22.533 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-26T06:37:22.534 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-26T06:37:22.534 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-03-26T06:37:22.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:22.675 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2024-03-26T06:37:22.675 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2024-03-26T06:37:22.675 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set brx.0 up 2024-03-26T06:37:22.675 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set dev brx.0 master ceph-brx 2024-03-26T06:37:22.675 DEBUG:teuthology.orchestra.run.smithi038:> ') 2024-03-26T06:37:22.717 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-26T06:37:22.717 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-26T06:37:22.772 INFO:teuthology.orchestra.run.smithi038.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-26T06:37:22.772 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-26T06:37:22.772 DEBUG:teuthology.orchestra.run.smithi038:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-26T06:37:22.826 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-26T06:37:22.826 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-26T06:37:22.827 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/a4829ff3-867c-4b07-aa33-9af29bd922f7 2024-03-26T06:37:22.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:22.827 DEBUG:teuthology.orchestra.run.smithi038:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/a4829ff3-867c-4b07-aa33-9af29bd922f7 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-26T06:37:22.912 INFO:teuthology.orchestra.run.smithi038.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-26T06:37:22.913 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-26T06:37:22.913 INFO:teuthology.orchestra.run.smithi038.stdout:invalid new device string format 2024-03-26T06:37:23.032 INFO:teuthology.orchestra.run.smithi038.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-26T06:37:23.032 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-26T06:37:23.032 INFO:teuthology.orchestra.run.smithi038.stdout:invalid new device string format 2024-03-26T06:37:23.033 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: resolved to: "172.21.15.38:6789,172.21.15.121:6789,172.21.15.170:6789" 2024-03-26T06:37:23.033 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.38:6789,172.21.15.121:6789,172.21.15.170:6789:/volumes/qa/sv_1/a4829ff3-867c-4b07-aa33-9af29bd922f7 2024-03-26T06:37:23.033 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=c2824556-eb39-11ee-95d0-87774f69a715" will pass to kernel 2024-03-26T06:37:23.033 INFO:teuthology.orchestra.run.smithi038.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-26T06:37:23.033 INFO:teuthology.orchestra.run.smithi038.stdout: You just mounted a file system that supports labels which does not 2024-03-26T06:37:23.033 INFO:teuthology.orchestra.run.smithi038.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-26T06:37:23.033 INFO:teuthology.orchestra.run.smithi038.stdout: applications will generate AVC messages and not be allowed access to 2024-03-26T06:37:23.033 INFO:teuthology.orchestra.run.smithi038.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-26T06:37:23.035 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-26T06:37:23.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:23.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-26T06:37:23.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:23.109 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-26T06:37:23.167 INFO:teuthology.orchestra.run.smithi038.stdout:client24941 2024-03-26T06:37:23.167 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:37:23.167 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-26T06:37:23.223 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:23.223 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-26T06:37:23.279 INFO:teuthology.orchestra.run.smithi038.stdout:client24941 2024-03-26T06:37:23.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:23.279 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-26T06:37:23.336 INFO:teuthology.orchestra.run.smithi038.stdout:client24941 2024-03-26T06:37:23.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:37:23.336 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/c2824556-eb39-11ee-95d0-87774f69a715.client24941/status') 2024-03-26T06:37:23.403 INFO:teuthology.orchestra.run.smithi038.stdout:instance: client.24941 (3)172.21.15.38:0/1509838523 2024-03-26T06:37:23.403 INFO:teuthology.orchestra.run.smithi038.stdout:blocklisted: false 2024-03-26T06:37:23.403 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2024-03-26T06:37:23.403 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2024-03-26T06:37:23.403 INFO:teuthology.orchestra.run.smithi038.stderr:71 bytes copied, 2.1449e-05 s, 3.3 MB/s 2024-03-26T06:37:23.404 INFO:teuthology.run_tasks:Running task exec... 2024-03-26T06:37:23.415 INFO:teuthology.task.exec:Executing custom commands... 2024-03-26T06:37:23.415 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T06:37:23.416 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-26T06:37:23.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.546+0000 7fda25b1b640 1 -- 172.21.15.38:0/831187553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda20101de0 msgr2=0x7fda201041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:23.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.546+0000 7fda25b1b640 1 --2- 172.21.15.38:0/831187553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda20101de0 0x7fda201041d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fda10009900 tx=0x7fda1002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:23.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.546+0000 7fda25b1b640 1 -- 172.21.15.38:0/831187553 shutdown_connections 2024-03-26T06:37:23.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.546+0000 7fda25b1b640 1 --2- 172.21.15.38:0/831187553 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda20107040 0x7fda20109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:23.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.546+0000 7fda25b1b640 1 --2- 172.21.15.38:0/831187553 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20104710 0x7fda20106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:23.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.546+0000 7fda25b1b640 1 --2- 172.21.15.38:0/831187553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda20101de0 0x7fda201041d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:23.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.546+0000 7fda25b1b640 1 -- 172.21.15.38:0/831187553 >> 172.21.15.38:0/831187553 conn(0x7fda200fb7d0 msgr2=0x7fda200fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:23.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.546+0000 7fda25b1b640 1 -- 172.21.15.38:0/831187553 shutdown_connections 2024-03-26T06:37:23.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.546+0000 7fda25b1b640 1 -- 172.21.15.38:0/831187553 wait complete. 2024-03-26T06:37:23.551 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda25b1b640 1 Processor -- start 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda25b1b640 1 -- start start 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda25b1b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20101de0 0x7fda20101750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda25b1b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda20104710 0x7fda200ffd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda25b1b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda20107040 0x7fda201002d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda25b1b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2010d050 con 0x7fda20107040 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda25b1b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2010ced0 con 0x7fda20101de0 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda25b1b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2010d1d0 con 0x7fda20104710 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda20107040 0x7fda201002d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda1ffff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda20107040 0x7fda201002d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42392/0 (socket says 172.21.15.38:42392) 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda1ffff640 1 -- 172.21.15.38:0/1121523960 learned_addr learned my addr 172.21.15.38:0/1121523960 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda1f7fe640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20101de0 0x7fda20101750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda1effd640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda20104710 0x7fda200ffd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda1ffff640 1 -- 172.21.15.38:0/1121523960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda20104710 msgr2=0x7fda200ffd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:23.552 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda1ffff640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda20104710 0x7fda200ffd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:23.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda1ffff640 1 -- 172.21.15.38:0/1121523960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20101de0 msgr2=0x7fda20101750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:23.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda1ffff640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20101de0 0x7fda20101750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:23.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda1ffff640 1 -- 172.21.15.38:0/1121523960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda20100b60 con 0x7fda20107040 2024-03-26T06:37:23.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.547+0000 7fda1effd640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda20104710 0x7fda200ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:37:23.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.548+0000 7fda1ffff640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda20107040 0x7fda201002d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fda1800b970 tx=0x7fda1800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:23.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.548+0000 7fda1cff9640 1 -- 172.21.15.38:0/1121523960 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18019070 con 0x7fda20107040 2024-03-26T06:37:23.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.548+0000 7fda1cff9640 1 -- 172.21.15.38:0/1121523960 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda1800cb40 con 0x7fda20107040 2024-03-26T06:37:23.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.548+0000 7fda25b1b640 1 -- 172.21.15.38:0/1121523960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda201a1700 con 0x7fda20107040 2024-03-26T06:37:23.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.548+0000 7fda1cff9640 1 -- 172.21.15.38:0/1121523960 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18007650 con 0x7fda20107040 2024-03-26T06:37:23.553 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.548+0000 7fda25b1b640 1 -- 172.21.15.38:0/1121523960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda201a1c50 con 0x7fda20107040 2024-03-26T06:37:23.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.549+0000 7fda25b1b640 1 -- 172.21.15.38:0/1121523960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda200fd2c0 con 0x7fda20107040 2024-03-26T06:37:23.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.550+0000 7fda1cff9640 1 -- 172.21.15.38:0/1121523960 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fda1801c070 con 0x7fda20107040 2024-03-26T06:37:23.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.550+0000 7fda1cff9640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda080a28a0 0x7fda080a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:23.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.550+0000 7fda1f7fe640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda080a28a0 0x7fda080a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:23.554 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.550+0000 7fda1cff9640 1 -- 172.21.15.38:0/1121523960 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fda180ce010 con 0x7fda20107040 2024-03-26T06:37:23.555 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.551+0000 7fda1f7fe640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda080a28a0 0x7fda080a4d60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fda1002c7b0 tx=0x7fda10005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:23.557 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.555+0000 7fda1cff9640 1 -- 172.21.15.38:0/1121523960 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fda1809aa60 con 0x7fda20107040 2024-03-26T06:37:23.780 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:23.778+0000 7fda25b1b640 1 -- 172.21.15.38:0/1121523960 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7fda20061ef0 con 0x7fda20107040 2024-03-26T06:37:23.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:23 smithi038 ceph-mon[32265]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:37:23.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:23 smithi121 ceph-mon[36349]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:37:23.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:23 smithi170 ceph-mon[36912]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:37:24.463 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.461+0000 7fda1cff9640 1 -- 172.21.15.38:0/1121523960 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7fda1809eb00 con 0x7fda20107040 2024-03-26T06:37:24.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.463+0000 7fda25b1b640 1 -- 172.21.15.38:0/1121523960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda080a28a0 msgr2=0x7fda080a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:24.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.463+0000 7fda25b1b640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda080a28a0 0x7fda080a4d60 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fda1002c7b0 tx=0x7fda10005fd0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.463+0000 7fda25b1b640 1 -- 172.21.15.38:0/1121523960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda20107040 msgr2=0x7fda201002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:24.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.463+0000 7fda25b1b640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda20107040 0x7fda201002d0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fda1800b970 tx=0x7fda1800be30 comp rx=0 tx=0).stop 2024-03-26T06:37:24.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.464+0000 7fda25b1b640 1 -- 172.21.15.38:0/1121523960 shutdown_connections 2024-03-26T06:37:24.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.464+0000 7fda25b1b640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fda080a28a0 0x7fda080a4d60 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.464+0000 7fda25b1b640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda20107040 0x7fda201002d0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.464+0000 7fda25b1b640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fda20104710 0x7fda200ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.464+0000 7fda25b1b640 1 --2- 172.21.15.38:0/1121523960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda20101de0 0x7fda20101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.464+0000 7fda25b1b640 1 -- 172.21.15.38:0/1121523960 >> 172.21.15.38:0/1121523960 conn(0x7fda200fb7d0 msgr2=0x7fda20102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:24.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.464+0000 7fda25b1b640 1 -- 172.21.15.38:0/1121523960 shutdown_connections 2024-03-26T06:37:24.467 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.464+0000 7fda25b1b640 1 -- 172.21.15.38:0/1121523960 wait complete. 2024-03-26T06:37:24.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-26T06:37:24.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.589+0000 7ff86b812640 1 -- 172.21.15.38:0/1915872227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff864101de0 msgr2=0x7ff8641041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.589+0000 7ff86b812640 1 --2- 172.21.15.38:0/1915872227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff864101de0 0x7ff8641041d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7ff860009900 tx=0x7ff86002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.589+0000 7ff86b812640 1 -- 172.21.15.38:0/1915872227 shutdown_connections 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.589+0000 7ff86b812640 1 --2- 172.21.15.38:0/1915872227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff864107040 0x7ff864109490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.589+0000 7ff86b812640 1 --2- 172.21.15.38:0/1915872227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff864104710 0x7ff864106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.589+0000 7ff86b812640 1 --2- 172.21.15.38:0/1915872227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff864101de0 0x7ff8641041d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.589+0000 7ff86b812640 1 -- 172.21.15.38:0/1915872227 >> 172.21.15.38:0/1915872227 conn(0x7ff8640fb7d0 msgr2=0x7ff8640fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.589+0000 7ff86b812640 1 -- 172.21.15.38:0/1915872227 shutdown_connections 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.589+0000 7ff86b812640 1 -- 172.21.15.38:0/1915872227 wait complete. 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff86b812640 1 Processor -- start 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff86b812640 1 -- start start 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff86b812640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff864101de0 0x7ff864197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:24.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff86b812640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff864104710 0x7ff864197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff86b812640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff864107040 0x7ff86419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff86b812640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86410cff0 con 0x7ff864101de0 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff86b812640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86410ce70 con 0x7ff864104710 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff86b812640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86410d170 con 0x7ff864107040 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff8695ae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff864101de0 0x7ff864197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff8695ae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff864101de0 0x7ff864197530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39464/0 (socket says 172.21.15.38:39464) 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff8695ae640 1 -- 172.21.15.38:0/1789323978 learned_addr learned my addr 172.21.15.38:0/1789323978 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff868dad640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff864104710 0x7ff864197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff8695ae640 1 -- 172.21.15.38:0/1789323978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff864107040 msgr2=0x7ff86419eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff8695ae640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff864107040 0x7ff86419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff8695ae640 1 -- 172.21.15.38:0/1789323978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff864104710 msgr2=0x7ff864197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff8695ae640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff864104710 0x7ff864197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff8695ae640 1 -- 172.21.15.38:0/1789323978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86419f150 con 0x7ff864101de0 2024-03-26T06:37:24.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff868dad640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff864104710 0x7ff864197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:24.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.590+0000 7ff8695ae640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff864101de0 0x7ff864197530 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff860009230 tx=0x7ff860004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:24.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.591+0000 7ff85a7fc640 1 -- 172.21.15.38:0/1789323978 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff860041070 con 0x7ff864101de0 2024-03-26T06:37:24.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.591+0000 7ff85a7fc640 1 -- 172.21.15.38:0/1789323978 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8600054e0 con 0x7ff864101de0 2024-03-26T06:37:24.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.591+0000 7ff85a7fc640 1 -- 172.21.15.38:0/1789323978 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8600346a0 con 0x7ff864101de0 2024-03-26T06:37:24.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.591+0000 7ff86b812640 1 -- 172.21.15.38:0/1789323978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86419f3e0 con 0x7ff864101de0 2024-03-26T06:37:24.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.591+0000 7ff86b812640 1 -- 172.21.15.38:0/1789323978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86419f8d0 con 0x7ff864101de0 2024-03-26T06:37:24.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.592+0000 7ff86b812640 1 -- 172.21.15.38:0/1789323978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8640fd2c0 con 0x7ff864101de0 2024-03-26T06:37:24.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.593+0000 7ff85a7fc640 1 -- 172.21.15.38:0/1789323978 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff860044070 con 0x7ff864101de0 2024-03-26T06:37:24.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.593+0000 7ff85a7fc640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8500a28a0 0x7ff8500a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:24.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.593+0000 7ff868dad640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8500a28a0 0x7ff8500a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:24.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.593+0000 7ff85a7fc640 1 -- 172.21.15.38:0/1789323978 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7ff8600f0bb0 con 0x7ff864101de0 2024-03-26T06:37:24.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.594+0000 7ff868dad640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8500a28a0 0x7ff8500a4d60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff864198a40 tx=0x7ff854005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:24.599 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.597+0000 7ff85a7fc640 1 -- 172.21.15.38:0/1789323978 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff8600bdc50 con 0x7ff864101de0 2024-03-26T06:37:24.809 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.807+0000 7ff86b812640 1 -- 172.21.15.38:0/1789323978 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8640617d0 con 0x7ff8500a28a0 2024-03-26T06:37:24.815 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:24 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1121523960' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-26T06:37:24.825 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.823+0000 7ff85a7fc640 1 -- 172.21.15.38:0/1789323978 <== mgr.14152 v2:172.21.15.38:6800/2716021457 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7ff8640617d0 con 0x7ff8500a28a0 2024-03-26T06:37:24.825 INFO:teuthology.orchestra.run.smithi038.stdout:{} 2024-03-26T06:37:24.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 -- 172.21.15.38:0/1789323978 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8500a28a0 msgr2=0x7ff8500a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:24.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8500a28a0 0x7ff8500a4d60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7ff864198a40 tx=0x7ff854005d30 comp rx=0 tx=0).stop 2024-03-26T06:37:24.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 -- 172.21.15.38:0/1789323978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff864101de0 msgr2=0x7ff864197530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:24.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff864101de0 0x7ff864197530 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7ff860009230 tx=0x7ff860004040 comp rx=0 tx=0).stop 2024-03-26T06:37:24.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 -- 172.21.15.38:0/1789323978 shutdown_connections 2024-03-26T06:37:24.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7ff8500a28a0 0x7ff8500a4d60 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff864107040 0x7ff86419eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff864104710 0x7ff864197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 --2- 172.21.15.38:0/1789323978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff864101de0 0x7ff864197530 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 -- 172.21.15.38:0/1789323978 >> 172.21.15.38:0/1789323978 conn(0x7ff8640fb7d0 msgr2=0x7ff864102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:24.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 -- 172.21.15.38:0/1789323978 shutdown_connections 2024-03-26T06:37:24.828 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.825+0000 7ff86b812640 1 -- 172.21.15.38:0/1789323978 wait complete. 2024-03-26T06:37:24.838 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-26T06:37:24.849 INFO:teuthology.run_tasks:Running task exec... 2024-03-26T06:37:24.885 INFO:teuthology.task.exec:Executing custom commands... 2024-03-26T06:37:24.886 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T06:37:24.886 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-26T06:37:24.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:24 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1121523960' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-26T06:37:24.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:24 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1121523960' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-26T06:37:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.986+0000 7f3feb68c640 1 -- 172.21.15.38:0/1619478291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe4101320 msgr2=0x7f3fe4101740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.986+0000 7f3feb68c640 1 --2- 172.21.15.38:0/1619478291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe4101320 0x7f3fe4101740 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f3fe0009900 tx=0x7f3fe002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.986+0000 7f3feb68c640 1 -- 172.21.15.38:0/1619478291 shutdown_connections 2024-03-26T06:37:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.986+0000 7f3feb68c640 1 --2- 172.21.15.38:0/1619478291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fe4102ed0 0x7f3fe4109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.986+0000 7f3feb68c640 1 --2- 172.21.15.38:0/1619478291 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fe4102510 0x7f3fe4102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.986+0000 7f3feb68c640 1 --2- 172.21.15.38:0/1619478291 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe4101320 0x7f3fe4101740 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.986+0000 7f3feb68c640 1 -- 172.21.15.38:0/1619478291 >> 172.21.15.38:0/1619478291 conn(0x7f3fe40fc8b0 msgr2=0x7f3fe40fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.986+0000 7f3feb68c640 1 -- 172.21.15.38:0/1619478291 shutdown_connections 2024-03-26T06:37:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.986+0000 7f3feb68c640 1 -- 172.21.15.38:0/1619478291 wait complete. 2024-03-26T06:37:24.992 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3feb68c640 1 Processor -- start 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3feb68c640 1 -- start start 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3feb68c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fe4101320 0x7f3fe41975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3feb68c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fe4102510 0x7f3fe4197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3feb68c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe4102ed0 0x7f3fe419eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3feb68c640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe410cff0 con 0x7f3fe4102ed0 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3feb68c640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe410ce70 con 0x7f3fe4101320 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3feb68c640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe410d170 con 0x7f3fe4102510 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3fe9c29640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe4102ed0 0x7f3fe419eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3fe9c29640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe4102ed0 0x7f3fe419eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39490/0 (socket says 172.21.15.38:39490) 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3fe9c29640 1 -- 172.21.15.38:0/2797878186 learned_addr learned my addr 172.21.15.38:0/2797878186 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3fe9428640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fe4101320 0x7f3fe41975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3fe8c27640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fe4102510 0x7f3fe4197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:24.993 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3fe9c29640 1 -- 172.21.15.38:0/2797878186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fe4102510 msgr2=0x7f3fe4197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3fe9c29640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fe4102510 0x7f3fe4197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3fe9c29640 1 -- 172.21.15.38:0/2797878186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fe4101320 msgr2=0x7f3fe41975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3fe9c29640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fe4101320 0x7f3fe41975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.987+0000 7f3fe9c29640 1 -- 172.21.15.38:0/2797878186 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe419f1f0 con 0x7f3fe4102ed0 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.988+0000 7f3fe9428640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fe4101320 0x7f3fe41975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.988+0000 7f3fe9c29640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe4102ed0 0x7f3fe419eb80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00eaa0 tx=0x7f3fdc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.988+0000 7f3fda7fc640 1 -- 172.21.15.38:0/2797878186 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc00cdb0 con 0x7f3fe4102ed0 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.988+0000 7f3fda7fc640 1 -- 172.21.15.38:0/2797878186 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3fdc005020 con 0x7f3fe4102ed0 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.988+0000 7f3feb68c640 1 -- 172.21.15.38:0/2797878186 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe419f4e0 con 0x7f3fe4102ed0 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.988+0000 7f3fda7fc640 1 -- 172.21.15.38:0/2797878186 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fdc0116b0 con 0x7f3fe4102ed0 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.988+0000 7f3feb68c640 1 -- 172.21.15.38:0/2797878186 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe419fa30 con 0x7f3fe4102ed0 2024-03-26T06:37:24.994 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.989+0000 7f3feb68c640 1 -- 172.21.15.38:0/2797878186 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe4108570 con 0x7f3fe4102ed0 2024-03-26T06:37:24.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.990+0000 7f3fda7fc640 1 -- 172.21.15.38:0/2797878186 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3fdc005560 con 0x7f3fe4102ed0 2024-03-26T06:37:24.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.990+0000 7f3fda7fc640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3fd00a2920 0x7f3fd00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:24.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.990+0000 7f3fe9428640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3fd00a2920 0x7f3fd00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:24.995 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.990+0000 7f3fda7fc640 1 -- 172.21.15.38:0/2797878186 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7f3fdc0ced00 con 0x7f3fe4102ed0 2024-03-26T06:37:24.996 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.991+0000 7f3fe9428640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3fd00a2920 0x7f3fd00a4de0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3fe002c7b0 tx=0x7f3fe0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:24.997 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:24.995+0000 7f3fda7fc640 1 -- 172.21.15.38:0/2797878186 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3fdc016030 con 0x7f3fe4102ed0 2024-03-26T06:37:25.083 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:24 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:24.811+0000 7f0f5ebf2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:25.083 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:24 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:24.811+0000 7f0f5ebf2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:25.083 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:24 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:24.811+0000 7f0f5ebf2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:25.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:24 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:24.811+0000 7f0f5ebf2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:25.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:24 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:24.811+0000 7f0f5ebf2700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:25.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.229+0000 7f3feb68c640 1 -- 172.21.15.38:0/2797878186 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f3fe4061ef0 con 0x7f3fe4102ed0 2024-03-26T06:37:25.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.467+0000 7f3fda7fc640 1 -- 172.21.15.38:0/2797878186 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f3fdc09bda0 con 0x7f3fe4102ed0 2024-03-26T06:37:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.469+0000 7f3feb68c640 1 -- 172.21.15.38:0/2797878186 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3fd00a2920 msgr2=0x7f3fd00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.469+0000 7f3feb68c640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3fd00a2920 0x7f3fd00a4de0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3fe002c7b0 tx=0x7f3fe0005fd0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.469+0000 7f3feb68c640 1 -- 172.21.15.38:0/2797878186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe4102ed0 msgr2=0x7f3fe419eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.469+0000 7f3feb68c640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe4102ed0 0x7f3fe419eb80 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f3fdc00eaa0 tx=0x7f3fdc00ef60 comp rx=0 tx=0).stop 2024-03-26T06:37:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.470+0000 7f3feb68c640 1 -- 172.21.15.38:0/2797878186 shutdown_connections 2024-03-26T06:37:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.470+0000 7f3feb68c640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7f3fd00a2920 0x7f3fd00a4de0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.472 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.470+0000 7f3feb68c640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3fe4102ed0 0x7f3fe419eb80 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.470+0000 7f3feb68c640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3fe4102510 0x7f3fe4197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.470+0000 7f3feb68c640 1 --2- 172.21.15.38:0/2797878186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fe4101320 0x7f3fe41975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.470+0000 7f3feb68c640 1 -- 172.21.15.38:0/2797878186 >> 172.21.15.38:0/2797878186 conn(0x7f3fe40fc8b0 msgr2=0x7f3fe4105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:25.473 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.470+0000 7f3feb68c640 1 -- 172.21.15.38:0/2797878186 shutdown_connections 2024-03-26T06:37:25.474 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.472+0000 7f3feb68c640 1 -- 172.21.15.38:0/2797878186 wait complete. 2024-03-26T06:37:25.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-26T06:37:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.601+0000 7feb07532640 1 -- 172.21.15.38:0/21778949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb00104760 msgr2=0x7feb00106b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.601+0000 7feb07532640 1 --2- 172.21.15.38:0/21778949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb00104760 0x7feb00106b50 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009900 tx=0x7feaf002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.601+0000 7feb07532640 1 -- 172.21.15.38:0/21778949 shutdown_connections 2024-03-26T06:37:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.601+0000 7feb07532640 1 --2- 172.21.15.38:0/21778949 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb00107090 0x7feb00109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.601+0000 7feb07532640 1 --2- 172.21.15.38:0/21778949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb00104760 0x7feb00106b50 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.601+0000 7feb07532640 1 --2- 172.21.15.38:0/21778949 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb00101e30 0x7feb00104220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb07532640 1 -- 172.21.15.38:0/21778949 >> 172.21.15.38:0/21778949 conn(0x7feb000fb780 msgr2=0x7feb000fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb07532640 1 -- 172.21.15.38:0/21778949 shutdown_connections 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb07532640 1 -- 172.21.15.38:0/21778949 wait complete. 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb07532640 1 Processor -- start 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb07532640 1 -- start start 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb07532640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb00101e30 0x7feb001975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb07532640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb00104760 0x7feb00197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb07532640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb00107090 0x7feb0019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb07532640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0010cfc0 con 0x7feb00101e30 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb07532640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0010ce40 con 0x7feb00104760 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb07532640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0010d140 con 0x7feb00107090 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb052ce640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb00101e30 0x7feb001975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb052ce640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb00101e30 0x7feb001975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39514/0 (socket says 172.21.15.38:39514) 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.602+0000 7feb052ce640 1 -- 172.21.15.38:0/2047292825 learned_addr learned my addr 172.21.15.38:0/2047292825 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feb04acd640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb00104760 0x7feb00197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feb05acf640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb00107090 0x7feb0019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feb052ce640 1 -- 172.21.15.38:0/2047292825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb00107090 msgr2=0x7feb0019eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feb052ce640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb00107090 0x7feb0019eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feb052ce640 1 -- 172.21.15.38:0/2047292825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb00104760 msgr2=0x7feb00197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feb052ce640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb00104760 0x7feb00197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feb052ce640 1 -- 172.21.15.38:0/2047292825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0019f1e0 con 0x7feb00101e30 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feb052ce640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb00101e30 0x7feb001975c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7feafc00e9f0 tx=0x7feafc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feaf67fc640 1 -- 172.21.15.38:0/2047292825 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc00cd80 con 0x7feb00101e30 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feaf67fc640 1 -- 172.21.15.38:0/2047292825 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feafc005500 con 0x7feb00101e30 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feaf67fc640 1 -- 172.21.15.38:0/2047292825 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feafc010690 con 0x7feb00101e30 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feb07532640 1 -- 172.21.15.38:0/2047292825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0019f4d0 con 0x7feb00101e30 2024-03-26T06:37:25.608 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.603+0000 7feb07532640 1 -- 172.21.15.38:0/2047292825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0019f9a0 con 0x7feb00101e30 2024-03-26T06:37:25.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.604+0000 7feb07532640 1 -- 172.21.15.38:0/2047292825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb000fd270 con 0x7feb00101e30 2024-03-26T06:37:25.609 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.605+0000 7feaf67fc640 1 -- 172.21.15.38:0/2047292825 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 137358+0+0 (secure 0 0 0) 0x7feafc0046c0 con 0x7feb00101e30 2024-03-26T06:37:25.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.606+0000 7feaf67fc640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7feaec0a2990 0x7feaec0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:25.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.606+0000 7feb04acd640 1 -- 172.21.15.38:0/2047292825 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7feaec0a2990 msgr2=0x7feaec0a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2716021457 2024-03-26T06:37:25.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.606+0000 7feaf67fc640 1 -- 172.21.15.38:0/2047292825 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7feafc0cf000 con 0x7feb00101e30 2024-03-26T06:37:25.610 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.606+0000 7feb04acd640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7feaec0a2990 0x7feaec0a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T06:37:25.613 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.611+0000 7feaf67fc640 1 -- 172.21.15.38:0/2047292825 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7feafc09bf80 con 0x7feb00101e30 2024-03-26T06:37:25.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:25 smithi038 ceph-mon[32265]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:37:25.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:25 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1121523960' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-26T06:37:25.775 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:25 smithi038 ceph-mon[32265]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:37:25.776 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:25 smithi038 ceph-mon[32265]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:25.776 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:25 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2797878186' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-26T06:37:25.776 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:25 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: ignoring --setuser ceph since I am not root 2024-03-26T06:37:25.776 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:25 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: ignoring --setgroup ceph since I am not root 2024-03-26T06:37:25.799 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.796+0000 7feb07532640 1 -- 172.21.15.38:0/2047292825 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7feb0002ce70 con 0x7feb00101e30 2024-03-26T06:37:25.803 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.800+0000 7feaf67fc640 1 -- 172.21.15.38:0/2047292825 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) v1 ==== 138+0+0 (secure 0 0 0) 0x7feafc09b880 con 0x7feb00101e30 2024-03-26T06:37:25.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 -- 172.21.15.38:0/2047292825 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7feaec0a2990 msgr2=0x7feaec0a4e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:37:25.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7feaec0a2990 0x7feaec0a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 -- 172.21.15.38:0/2047292825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb00101e30 msgr2=0x7feb001975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:25.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb00101e30 0x7feb001975c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7feafc00e9f0 tx=0x7feafc00edf0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 -- 172.21.15.38:0/2047292825 shutdown_connections 2024-03-26T06:37:25.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7feaec0a2990 0x7feaec0a4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7feb00107090 0x7feb0019eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.806 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb00104760 0x7feb00197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 --2- 172.21.15.38:0/2047292825 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feb00101e30 0x7feb001975c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 -- 172.21.15.38:0/2047292825 >> 172.21.15.38:0/2047292825 conn(0x7feb000fb780 msgr2=0x7feb00102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 -- 172.21.15.38:0/2047292825 shutdown_connections 2024-03-26T06:37:25.807 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.802+0000 7feb07532640 1 -- 172.21.15.38:0/2047292825 wait complete. 2024-03-26T06:37:25.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-26T06:37:25.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:25 smithi121 ceph-mon[36349]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:37:25.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:25 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1121523960' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-26T06:37:25.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:25 smithi121 ceph-mon[36349]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:37:25.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:25 smithi121 ceph-mon[36349]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:25.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:25 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2797878186' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-26T06:37:25.896 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:25 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: ignoring --setuser ceph since I am not root 2024-03-26T06:37:25.896 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:25 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: ignoring --setgroup ceph since I am not root 2024-03-26T06:37:25.896 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:25 smithi121 ceph-mgr[37738]: -- 172.21.15.121:0/3550206109 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5626f97505a0 con 0x5626f9724c00 2024-03-26T06:37:25.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:25 smithi170 ceph-mon[36912]: pgmap v347: 97 pgs: 97 active+clean; 639 KiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:37:25.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:25 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1121523960' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-26T06:37:25.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:25 smithi170 ceph-mon[36912]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:37:25.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:25 smithi170 ceph-mon[36912]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:25.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:25 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2797878186' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-26T06:37:25.918 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:25 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: ignoring --setuser ceph since I am not root 2024-03-26T06:37:25.918 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:25 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: ignoring --setgroup ceph since I am not root 2024-03-26T06:37:25.918 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:25 smithi170 ceph-mgr[38777]: -- 172.21.15.170:0/1033074723 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fb2ef945a0 con 0x55fb2ef69000 2024-03-26T06:37:25.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.925+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4001150476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb44100580 msgr2=0x7fdb441009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:25.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.925+0000 7fdb4b50a640 1 --2- 172.21.15.38:0/4001150476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb44100580 0x7fdb441009e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdb34009900 tx=0x7fdb3402bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:25.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.925+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4001150476 shutdown_connections 2024-03-26T06:37:25.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.925+0000 7fdb4b50a640 1 --2- 172.21.15.38:0/4001150476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb44100f20 0x7fdb441013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.925+0000 7fdb4b50a640 1 --2- 172.21.15.38:0/4001150476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb44100580 0x7fdb441009e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.925+0000 7fdb4b50a640 1 --2- 172.21.15.38:0/4001150476 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb440ffc20 0x7fdb44100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.925+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4001150476 >> 172.21.15.38:0/4001150476 conn(0x7fdb440fb7c0 msgr2=0x7fdb440fdc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:25.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.925+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4001150476 shutdown_connections 2024-03-26T06:37:25.931 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.925+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4001150476 wait complete. 2024-03-26T06:37:25.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb4b50a640 1 Processor -- start 2024-03-26T06:37:25.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb4b50a640 1 -- start start 2024-03-26T06:37:25.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb4b50a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb440ffc20 0x7fdb44199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:25.932 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb4b50a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb44100580 0x7fdb44199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb4b50a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb44100f20 0x7fdb441a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb4b50a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4410f1b0 con 0x7fdb440ffc20 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb4b50a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4410f030 con 0x7fdb44100f20 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb4b50a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4410f330 con 0x7fdb44100580 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb492a6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb440ffc20 0x7fdb44199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb48aa5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb44100580 0x7fdb44199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb49aa7640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb44100f20 0x7fdb441a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb48aa5640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb44100580 0x7fdb44199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:37352/0 (socket says 172.21.15.38:37352) 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb492a6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb440ffc20 0x7fdb44199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39530/0 (socket says 172.21.15.38:39530) 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.926+0000 7fdb48aa5640 1 -- 172.21.15.38:0/4231840247 learned_addr learned my addr 172.21.15.38:0/4231840247 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb49aa7640 1 -- 172.21.15.38:0/4231840247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb44100580 msgr2=0x7fdb44199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb49aa7640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb44100580 0x7fdb44199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb49aa7640 1 -- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb440ffc20 msgr2=0x7fdb44199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:25.933 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb49aa7640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb440ffc20 0x7fdb44199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb49aa7640 1 -- 172.21.15.38:0/4231840247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb441a13d0 con 0x7fdb44100f20 2024-03-26T06:37:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb492a6640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb440ffc20 0x7fdb44199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:37:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb49aa7640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb44100f20 0x7fdb441a0d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00da10 tx=0x7fdb3c00ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb3a7fc640 1 -- 172.21.15.38:0/4231840247 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c00bd70 con 0x7fdb44100f20 2024-03-26T06:37:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb3a7fc640 1 -- 172.21.15.38:0/4231840247 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb3c003e50 con 0x7fdb44100f20 2024-03-26T06:37:25.934 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4231840247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb441a16c0 con 0x7fdb44100f20 2024-03-26T06:37:25.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb3a7fc640 1 -- 172.21.15.38:0/4231840247 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3c0106a0 con 0x7fdb44100f20 2024-03-26T06:37:25.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.927+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4231840247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb441a1be0 con 0x7fdb44100f20 2024-03-26T06:37:25.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.928+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4231840247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4410a970 con 0x7fdb44100f20 2024-03-26T06:37:25.935 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.929+0000 7fdb3a7fc640 1 -- 172.21.15.38:0/4231840247 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 137358+0+0 (secure 0 0 0) 0x7fdb3c003970 con 0x7fdb44100f20 2024-03-26T06:37:25.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.929+0000 7fdb3a7fc640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb300a2a10 0x7fdb300a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:25.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.930+0000 7fdb492a6640 1 -- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb300a2a10 msgr2=0x7fdb300a4ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2716021457 2024-03-26T06:37:25.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.930+0000 7fdb3a7fc640 1 -- 172.21.15.38:0/4231840247 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fdb3c0cefb0 con 0x7fdb44100f20 2024-03-26T06:37:25.936 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.930+0000 7fdb492a6640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb300a2a10 0x7fdb300a4ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T06:37:25.938 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:25.934+0000 7fdb3a7fc640 1 -- 172.21.15.38:0/4231840247 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdb3c09bef0 con 0x7fdb44100f20 2024-03-26T06:37:26.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.131+0000 7fdb492a6640 1 -- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb300a2a10 msgr2=0x7fdb300a4ed0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2716021457 2024-03-26T06:37:26.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.131+0000 7fdb492a6640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb300a2a10 0x7fdb300a4ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T06:37:26.148 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.144+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4231840247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fdb4402ce70 con 0x7fdb44100f20 2024-03-26T06:37:26.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.154+0000 7fdb3a7fc640 1 -- 172.21.15.38:0/4231840247 <== mon.2 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) v1 ==== 133+0+0 (secure 0 0 0) 0x7fdb3c09b7f0 con 0x7fdb44100f20 2024-03-26T06:37:26.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb300a2a10 msgr2=0x7fdb300a4ed0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:37:26.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb300a2a10 0x7fdb300a4ed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:26.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4231840247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb44100f20 msgr2=0x7fdb441a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:26.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb44100f20 0x7fdb441a0d60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c00da10 tx=0x7fdb3c00ded0 comp rx=0 tx=0).stop 2024-03-26T06:37:26.160 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4231840247 shutdown_connections 2024-03-26T06:37:26.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fdb300a2a10 0x7fdb300a4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:26.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb44100f20 0x7fdb441a0d60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:26.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fdb44100580 0x7fdb44199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:26.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 --2- 172.21.15.38:0/4231840247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdb440ffc20 0x7fdb44199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:26.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4231840247 >> 172.21.15.38:0/4231840247 conn(0x7fdb440fb7c0 msgr2=0x7fdb440fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:26.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4231840247 shutdown_connections 2024-03-26T06:37:26.161 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.156+0000 7fdb4b50a640 1 -- 172.21.15.38:0/4231840247 wait complete. 2024-03-26T06:37:26.171 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-26T06:37:26.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.310+0000 7fd27d7cd640 1 -- 172.21.15.38:0/3934630672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd278102530 msgr2=0x7fd2781029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:26.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.310+0000 7fd27d7cd640 1 --2- 172.21.15.38:0/3934630672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd278102530 0x7fd2781029b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd26c009860 tx=0x7fd26c02bb70 comp rx=0 tx=0).stop 2024-03-26T06:37:26.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.310+0000 7fd27d7cd640 1 -- 172.21.15.38:0/3934630672 shutdown_connections 2024-03-26T06:37:26.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.310+0000 7fd27d7cd640 1 --2- 172.21.15.38:0/3934630672 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd278102ef0 0x7fd278109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:26.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.310+0000 7fd27d7cd640 1 --2- 172.21.15.38:0/3934630672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd278102530 0x7fd2781029b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:26.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.310+0000 7fd27d7cd640 1 --2- 172.21.15.38:0/3934630672 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd278101340 0x7fd278101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:26.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.310+0000 7fd27d7cd640 1 -- 172.21.15.38:0/3934630672 >> 172.21.15.38:0/3934630672 conn(0x7fd2780fc8b0 msgr2=0x7fd2780fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:26.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.310+0000 7fd27d7cd640 1 -- 172.21.15.38:0/3934630672 shutdown_connections 2024-03-26T06:37:26.314 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.310+0000 7fd27d7cd640 1 -- 172.21.15.38:0/3934630672 wait complete. 2024-03-26T06:37:26.315 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd27d7cd640 1 Processor -- start 2024-03-26T06:37:26.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd27d7cd640 1 -- start start 2024-03-26T06:37:26.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd27d7cd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd278101340 0x7fd278197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:26.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd27d7cd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd278102530 0x7fd278197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:26.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd27d7cd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd278102ef0 0x7fd27819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:26.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd27d7cd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27810cfd0 con 0x7fd278101340 2024-03-26T06:37:26.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd27d7cd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27810ce50 con 0x7fd278102530 2024-03-26T06:37:26.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd27d7cd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27810d150 con 0x7fd278102ef0 2024-03-26T06:37:26.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd276ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd278101340 0x7fd278197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:26.317 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd2777fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd278102ef0 0x7fd27819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd2777fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd278102ef0 0x7fd27819ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:37364/0 (socket says 172.21.15.38:37364) 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd2767fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd278102530 0x7fd278197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd276ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd278101340 0x7fd278197600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39558/0 (socket says 172.21.15.38:39558) 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.311+0000 7fd2777fe640 1 -- 172.21.15.38:0/2793747960 learned_addr learned my addr 172.21.15.38:0/2793747960 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd2767fc640 1 -- 172.21.15.38:0/2793747960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd278102ef0 msgr2=0x7fd27819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd2767fc640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd278102ef0 0x7fd27819ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd2767fc640 1 -- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd278101340 msgr2=0x7fd278197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd2767fc640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd278101340 0x7fd278197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd2767fc640 1 -- 172.21.15.38:0/2793747960 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd27819f220 con 0x7fd278102530 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd276ffd640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd278101340 0x7fd278197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd2777fe640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd278102ef0 0x7fd27819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:37:26.318 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd2767fc640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd278102530 0x7fd278197b40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd26c02c0b0 tx=0x7fd26c032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:26.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c041020 con 0x7fd278102530 2024-03-26T06:37:26.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd26c02c9f0 con 0x7fd278102530 2024-03-26T06:37:26.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd27d7cd640 1 -- 172.21.15.38:0/2793747960 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd278073520 con 0x7fd278102530 2024-03-26T06:37:26.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd26c039040 con 0x7fd278102530 2024-03-26T06:37:26.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.312+0000 7fd27d7cd640 1 -- 172.21.15.38:0/2793747960 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd278073a10 con 0x7fd278102530 2024-03-26T06:37:26.319 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.313+0000 7fd27d7cd640 1 -- 172.21.15.38:0/2793747960 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd278108660 con 0x7fd278102530 2024-03-26T06:37:26.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.314+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 19) v1 ==== 137358+0+0 (secure 0 0 0) 0x7fd26c044020 con 0x7fd278102530 2024-03-26T06:37:26.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.315+0000 7fd25ffff640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 0x7fd2600a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:26.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.315+0000 7fd276ffd640 1 -- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 msgr2=0x7fd2600a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2716021457 2024-03-26T06:37:26.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.315+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8240+0+0 (secure 0 0 0) 0x7fd26c0f06c0 con 0x7fd278102530 2024-03-26T06:37:26.320 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.315+0000 7fd276ffd640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 0x7fd2600a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T06:37:26.321 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.318+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd26c04b020 con 0x7fd278102530 2024-03-26T06:37:26.395 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:26.031+0000 7f7bd62b5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T06:37:26.395 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:26.125+0000 7f7bd62b5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T06:37:26.417 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:26 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:26.037+0000 7fe4da6b9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T06:37:26.418 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:26 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:26.131+0000 7fe4da6b9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T06:37:26.488 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:26 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:26.117+0000 7f651eb5a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T06:37:26.488 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:26 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:26.219+0000 7f651eb5a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T06:37:26.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.516+0000 7fd276ffd640 1 -- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 msgr2=0x7fd2600a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2716021457 2024-03-26T06:37:26.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.516+0000 7fd276ffd640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 0x7fd2600a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T06:37:26.532 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.528+0000 7fd27d7cd640 1 -- 172.21.15.38:0/2793747960 --> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd278074e60 con 0x7fd2600a2990 2024-03-26T06:37:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:26 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2797878186' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-26T06:37:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:26 smithi121 ceph-mon[36349]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-26T06:37:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:26 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2047292825' entity='client.admin' 2024-03-26T06:37:26.766 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:26 smithi121 ceph-mon[36349]: from='client.? ' entity='client.admin' 2024-03-26T06:37:26.768 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:26 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2797878186' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-26T06:37:26.768 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:26 smithi170 ceph-mon[36912]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-26T06:37:26.768 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:26 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2047292825' entity='client.admin' 2024-03-26T06:37:26.768 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:26 smithi170 ceph-mon[36912]: from='client.? ' entity='client.admin' 2024-03-26T06:37:26.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:26 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2797878186' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-26T06:37:26.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:26 smithi038 ceph-mon[32265]: mgrmap e19: x(active, since 8m), standbys: z, y 2024-03-26T06:37:26.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:26 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2047292825' entity='client.admin' 2024-03-26T06:37:26.808 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:26 smithi038 ceph-mon[32265]: from='client.? ' entity='client.admin' 2024-03-26T06:37:26.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.917+0000 7fd276ffd640 1 -- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 msgr2=0x7fd2600a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2716021457 2024-03-26T06:37:26.920 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:26.917+0000 7fd276ffd640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 0x7fd2600a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T06:37:27.083 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:26 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:26.895+0000 7f651eb5a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T06:37:27.083 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:26 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:26.991+0000 7f651eb5a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T06:37:27.093 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:26.762+0000 7f7bd62b5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T06:37:27.093 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:26.855+0000 7f7bd62b5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T06:37:27.093 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:26.995+0000 7f7bd62b5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T06:37:27.093 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:27.090+0000 7f7bd62b5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T06:37:27.094 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:26 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:26.766+0000 7fe4da6b9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T06:37:27.094 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:26 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:26.857+0000 7fe4da6b9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T06:37:27.094 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:26 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:26.998+0000 7fe4da6b9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T06:37:27.094 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:27.091+0000 7fe4da6b9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T06:37:27.395 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:27.175+0000 7f7bd62b5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T06:37:27.417 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:27.178+0000 7fe4da6b9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T06:37:27.583 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:27 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:27.138+0000 7f651eb5a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T06:37:27.583 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:27 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:27.237+0000 7f651eb5a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T06:37:27.583 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:27 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:27.323+0000 7f651eb5a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T06:37:27.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:27.718+0000 7fd276ffd640 1 -- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 msgr2=0x7fd2600a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2716021457 2024-03-26T06:37:27.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:27.718+0000 7fd276ffd640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 0x7fd2600a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T06:37:27.726 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:27.436+0000 7f7bd62b5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T06:37:27.726 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:27.544+0000 7f7bd62b5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T06:37:27.726 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:27.629+0000 7f7bd62b5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T06:37:27.727 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:27.723+0000 7f7bd62b5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T06:37:27.732 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:27.443+0000 7fe4da6b9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T06:37:27.732 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:27.551+0000 7fe4da6b9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T06:37:27.732 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:27.636+0000 7fe4da6b9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T06:37:27.732 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:27.729+0000 7fe4da6b9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T06:37:27.895 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:27 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:27.593+0000 7f651eb5a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T06:37:27.895 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:27 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:27.705+0000 7f651eb5a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T06:37:27.896 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:27 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:27.794+0000 7f651eb5a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T06:37:27.896 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:27 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:27.891+0000 7f651eb5a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T06:37:28.145 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:27.896+0000 7f7bd62b5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T06:37:28.167 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:27.905+0000 7fe4da6b9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T06:37:28.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:28 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:28.071+0000 7f651eb5a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T06:37:29.145 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:28 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:28.853+0000 7f7bd62b5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T06:37:29.145 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:28 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:28.982+0000 7f7bd62b5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T06:37:29.167 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:28 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:28.864+0000 7fe4da6b9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T06:37:29.168 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:28 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:28.993+0000 7fe4da6b9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T06:37:29.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:29.319+0000 7fd276ffd640 1 -- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 msgr2=0x7fd2600a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2716021457 2024-03-26T06:37:29.323 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:29.319+0000 7fd276ffd640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 0x7fd2600a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T06:37:29.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:29.069+0000 7f651eb5a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T06:37:29.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:29.204+0000 7f651eb5a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T06:37:29.598 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:29 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:29.311+0000 7f7bd62b5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T06:37:29.598 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:29 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:29.401+0000 7f7bd62b5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T06:37:29.612 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:29.326+0000 7fe4da6b9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T06:37:29.612 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:29.415+0000 7fe4da6b9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T06:37:29.833 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:29.545+0000 7f651eb5a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T06:37:29.833 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:29.639+0000 7f651eb5a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T06:37:29.895 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:29 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:29.595+0000 7f7bd62b5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:37:29.895 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:29 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:29.778+0000 7f7bd62b5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T06:37:29.917 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:29.609+0000 7fe4da6b9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:37:29.918 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:29.793+0000 7fe4da6b9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T06:37:30.322 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:30 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:30.061+0000 7f7bd62b5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:37:30.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:29.843+0000 7f651eb5a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:37:30.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:30 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:30.036+0000 7f651eb5a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T06:37:30.338 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:30 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:30.075+0000 7fe4da6b9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:37:30.598 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:30 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:30.333+0000 7f651eb5a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T06:37:30.645 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:30 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:30.319+0000 7f7bd62b5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T06:37:30.645 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:30 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:30.554+0000 7f7bd62b5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T06:37:30.667 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:30 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:30.336+0000 7fe4da6b9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T06:37:30.667 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:30 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:30.570+0000 7fe4da6b9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T06:37:31.083 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:30 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:30.594+0000 7f651eb5a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T06:37:31.083 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:30 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:30.840+0000 7f651eb5a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T06:37:31.645 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:31 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:31.349+0000 7f7bd62b5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T06:37:31.667 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:31 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:31.368+0000 7fe4da6b9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T06:37:32.054 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:31 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:31.682+0000 7f651eb5a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T06:37:32.228 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:31 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:31.915+0000 7f7bd62b5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T06:37:32.228 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:32.011+0000 7f7bd62b5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T06:37:32.252 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:31 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:31.937+0000 7fe4da6b9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T06:37:32.252 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:32.035+0000 7fe4da6b9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T06:37:32.302 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:32.299+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 <== mon.2 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 20) v1 ==== 137358+0+0 (secure 0 0 0) 0x7fd26c032640 con 0x7fd278102530 2024-03-26T06:37:32.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.271+0000 7f651eb5a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T06:37:32.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: Standby manager daemon y restarted 2024-03-26T06:37:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: Standby manager daemon y started 2024-03-26T06:37:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T06:37:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:37:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T06:37:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:37:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: Standby manager daemon z restarted 2024-03-26T06:37:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: Standby manager daemon z started 2024-03-26T06:37:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T06:37:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:37:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T06:37:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-mon[32265]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:37:32.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:32.524+0000 7fd276ffd640 1 -- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 msgr2=0x7fd2600a4e50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/2716021457 2024-03-26T06:37:32.527 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:32.524+0000 7fd276ffd640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 0x7fd2600a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-26T06:37:32.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:32.630+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 <== mon.2 v2:172.21.15.121:3300/0 8 ==== mgrmap(e 21) v1 ==== 136975+0+0 (secure 0 0 0) 0x7fd26c0b9680 con 0x7fd278102530 2024-03-26T06:37:32.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:32.630+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 msgr2=0x7fd2600a4e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:37:32.633 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:32.630+0000 7fd25ffff640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/2716021457,v1:172.21.15.38:6801/2716021457] conn(0x7fd2600a2990 0x7fd2600a4e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:32.645 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y[37734]: 2024-03-26T06:37:32.224+0000 7f7bd62b5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T06:37:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: Standby manager daemon y restarted 2024-03-26T06:37:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: Standby manager daemon y started 2024-03-26T06:37:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T06:37:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:37:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T06:37:32.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:37:32.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: Standby manager daemon z restarted 2024-03-26T06:37:32.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: Standby manager daemon z started 2024-03-26T06:37:32.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T06:37:32.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:37:32.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T06:37:32.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:32 smithi121 ceph-mon[36349]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:37:32.667 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z[38773]: 2024-03-26T06:37:32.250+0000 7fe4da6b9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: Standby manager daemon y restarted 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: Standby manager daemon y started 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.121:0/1680312309' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: Standby manager daemon z restarted 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: Standby manager daemon z started 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-26T06:37:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:32 smithi170 ceph-mon[36912]: from='mgr.? 172.21.15.170:0/3033855512' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T06:37:32.806 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.379+0000 7f651eb5a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T06:37:32.806 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.604+0000 7f651eb5a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T06:37:33.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.856+0000 7f6481602700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:33.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.856+0000 7f6481602700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:33.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.856+0000 7f6481602700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:33.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.856+0000 7f6481602700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:33.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.856+0000 7f6481602700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:33.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.857+0000 7f647cdf9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:33.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.857+0000 7f647cdf9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:33.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.857+0000 7f647cdf9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:33.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.857+0000 7f647cdf9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:33.084 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:37:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:37:32.857+0000 7f647cdf9700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:37:33.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-26T06:37:33.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: Active manager daemon x restarted 2024-03-26T06:37:33.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: Activating manager daemon x 2024-03-26T06:37:33.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: osdmap e89: 12 total, 12 up, 12 in 2024-03-26T06:37:33.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: mgrmap e21: x(active, starting, since 0.0109703s), standbys: z, y 2024-03-26T06:37:33.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-26T06:37:33.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:37:33.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:37:33.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:37:33.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:37:33.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:37:33.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T06:37:33.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T06:37:33.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T06:37:33.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: Manager daemon x is now available 2024-03-26T06:37:33.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:37:33.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:37:33.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T06:37:33.587 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:33 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T06:37:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.638+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 <== mon.2 v2:172.21.15.121:3300/0 9 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7fd26c0c5320 con 0x7fd278102530 2024-03-26T06:37:33.641 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.638+0000 7fd25ffff640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd2600355f0 0x7fd2600359d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:33.642 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.638+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 --> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd26c0bbd00 con 0x7fd2600355f0 2024-03-26T06:37:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-26T06:37:33.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: Active manager daemon x restarted 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: Activating manager daemon x 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: osdmap e89: 12 total, 12 up, 12 in 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: mgrmap e21: x(active, starting, since 0.0109703s), standbys: z, y 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-26T06:37:33.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:37:33.647 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: Manager daemon x is now available 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:37:33.648 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:37:33.649 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T06:37:33.649 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:33 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T06:37:33.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.643+0000 7fd276ffd640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd2600355f0 0x7fd2600359d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:33.649 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.647+0000 7fd276ffd640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd2600355f0 0x7fd2600359d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd278102390 tx=0x7fd268005f90 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:33.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-26T06:37:33.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: Active manager daemon x restarted 2024-03-26T06:37:33.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: Activating manager daemon x 2024-03-26T06:37:33.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: osdmap e89: 12 total, 12 up, 12 in 2024-03-26T06:37:33.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: mgrmap e21: x(active, starting, since 0.0109703s), standbys: z, y 2024-03-26T06:37:33.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T06:37:33.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T06:37:33.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-26T06:37:33.670 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T06:37:33.671 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T06:37:33.671 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T06:37:33.671 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: Manager daemon x is now available 2024-03-26T06:37:33.671 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:37:33.671 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:37:33.671 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T06:37:33.671 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:33 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T06:37:33.716 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.713+0000 7fd25ffff640 1 -- 172.21.15.38:0/2793747960 <== mgr.24883 v2:172.21.15.38:6800/1823987770 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fd26c0bbd00 con 0x7fd2600355f0 2024-03-26T06:37:33.716 INFO:teuthology.orchestra.run.smithi038.stdout:Schedule set for path / 2024-03-26T06:37:33.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.715+0000 7fd27d7cd640 1 -- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd2600355f0 msgr2=0x7fd2600359d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:33.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.715+0000 7fd27d7cd640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd2600355f0 0x7fd2600359d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd278102390 tx=0x7fd268005f90 comp rx=0 tx=0).stop 2024-03-26T06:37:33.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.715+0000 7fd27d7cd640 1 -- 172.21.15.38:0/2793747960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd278102530 msgr2=0x7fd278197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:33.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.715+0000 7fd27d7cd640 1 --2- 172.21.15.38:0/2793747960 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd278102530 0x7fd278197b40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd26c02c0b0 tx=0x7fd26c032430 comp rx=0 tx=0).stop 2024-03-26T06:37:33.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.715+0000 7fd2777fe640 1 -- 172.21.15.38:0/2793747960 reap_dead start 2024-03-26T06:37:33.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.715+0000 7fd27d7cd640 1 -- 172.21.15.38:0/2793747960 shutdown_connections 2024-03-26T06:37:33.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.715+0000 7fd27d7cd640 1 -- 172.21.15.38:0/2793747960 >> 172.21.15.38:0/2793747960 conn(0x7fd2780fc8b0 msgr2=0x7fd278105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:33.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.715+0000 7fd27d7cd640 1 -- 172.21.15.38:0/2793747960 shutdown_connections 2024-03-26T06:37:33.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.715+0000 7fd27d7cd640 1 -- 172.21.15.38:0/2793747960 wait complete. 2024-03-26T06:37:33.729 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-26T06:37:33.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.833+0000 7ff683875640 1 -- 172.21.15.38:0/1323327882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff67c0752b0 msgr2=0x7ff67c073700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:33.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.833+0000 7ff683875640 1 --2- 172.21.15.38:0/1323327882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff67c0752b0 0x7ff67c073700 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff678009900 tx=0x7ff67802bcd0 comp rx=0 tx=0).stop 2024-03-26T06:37:33.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.833+0000 7ff683875640 1 -- 172.21.15.38:0/1323327882 shutdown_connections 2024-03-26T06:37:33.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.833+0000 7ff683875640 1 --2- 172.21.15.38:0/1323327882 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff67c1091f0 0x7ff67c10b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:33.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.833+0000 7ff683875640 1 --2- 172.21.15.38:0/1323327882 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67c073cd0 0x7ff67c074130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:33.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.833+0000 7ff683875640 1 --2- 172.21.15.38:0/1323327882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff67c0752b0 0x7ff67c073700 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:33.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.833+0000 7ff683875640 1 -- 172.21.15.38:0/1323327882 >> 172.21.15.38:0/1323327882 conn(0x7ff67c0fc8b0 msgr2=0x7ff67c0fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.833+0000 7ff683875640 1 -- 172.21.15.38:0/1323327882 shutdown_connections 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.833+0000 7ff683875640 1 -- 172.21.15.38:0/1323327882 wait complete. 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff683875640 1 Processor -- start 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff683875640 1 -- start start 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff683875640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff67c073cd0 0x7ff67c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff683875640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff67c0752b0 0x7ff67c199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff683875640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67c1091f0 0x7ff67c1a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff683875640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c10f1a0 con 0x7ff67c0752b0 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff683875640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c10f020 con 0x7ff67c1091f0 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff683875640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67c10f320 con 0x7ff67c073cd0 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff680e10640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff67c0752b0 0x7ff67c199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff680e10640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff67c0752b0 0x7ff67c199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39726/0 (socket says 172.21.15.38:39726) 2024-03-26T06:37:33.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff680e10640 1 -- 172.21.15.38:0/3920041839 learned_addr learned my addr 172.21.15.38:0/3920041839 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:33.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff681e12640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67c1091f0 0x7ff67c1a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:33.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff681611640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff67c073cd0 0x7ff67c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:33.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff680e10640 1 -- 172.21.15.38:0/3920041839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff67c073cd0 msgr2=0x7ff67c1997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:33.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff680e10640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff67c073cd0 0x7ff67c1997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:33.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff680e10640 1 -- 172.21.15.38:0/3920041839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67c1091f0 msgr2=0x7ff67c1a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:33.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff680e10640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67c1091f0 0x7ff67c1a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:33.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff680e10640 1 -- 172.21.15.38:0/3920041839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c1a13c0 con 0x7ff67c0752b0 2024-03-26T06:37:33.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.834+0000 7ff680e10640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff67c0752b0 0x7ff67c199d30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00e9f0 tx=0x7ff66c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:33.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.835+0000 7ff6727fc640 1 -- 172.21.15.38:0/3920041839 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c00cd80 con 0x7ff67c0752b0 2024-03-26T06:37:33.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.835+0000 7ff6727fc640 1 -- 172.21.15.38:0/3920041839 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff66c010910 con 0x7ff67c0752b0 2024-03-26T06:37:33.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.835+0000 7ff683875640 1 -- 172.21.15.38:0/3920041839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff67c1a16b0 con 0x7ff67c0752b0 2024-03-26T06:37:33.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.835+0000 7ff6727fc640 1 -- 172.21.15.38:0/3920041839 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c018990 con 0x7ff67c0752b0 2024-03-26T06:37:33.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.835+0000 7ff683875640 1 -- 172.21.15.38:0/3920041839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67c1a1c50 con 0x7ff67c0752b0 2024-03-26T06:37:33.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.836+0000 7ff683875640 1 -- 172.21.15.38:0/3920041839 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67c107fb0 con 0x7ff67c0752b0 2024-03-26T06:37:33.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.837+0000 7ff6727fc640 1 -- 172.21.15.38:0/3920041839 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7ff66c010ac0 con 0x7ff67c0752b0 2024-03-26T06:37:33.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.837+0000 7ff6727fc640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6680a2730 0x7ff6680a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:33.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.837+0000 7ff681611640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6680a2730 0x7ff6680a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:33.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.837+0000 7ff6727fc640 1 -- 172.21.15.38:0/3920041839 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7ff66c0ceed0 con 0x7ff67c0752b0 2024-03-26T06:37:33.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.838+0000 7ff681611640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6680a2730 0x7ff6680a4bf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff678006010 tx=0x7ff67802c940 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:33.847 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:33.845+0000 7ff6727fc640 1 -- 172.21.15.38:0/3920041839 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff66c09c130 con 0x7ff67c0752b0 2024-03-26T06:37:34.057 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.055+0000 7ff683875640 1 -- 172.21.15.38:0/3920041839 --> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7ff67c061890 con 0x7ff6680a2730 2024-03-26T06:37:34.115 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.112+0000 7ff6727fc640 1 -- 172.21.15.38:0/3920041839 <== mgr.24883 v2:172.21.15.38:6800/1823987770 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7ff67c061890 con 0x7ff6680a2730 2024-03-26T06:37:34.115 INFO:teuthology.orchestra.run.smithi038.stdout:Retention added to path / 2024-03-26T06:37:34.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 -- 172.21.15.38:0/3920041839 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6680a2730 msgr2=0x7ff6680a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6680a2730 0x7ff6680a4bf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff678006010 tx=0x7ff67802c940 comp rx=0 tx=0).stop 2024-03-26T06:37:34.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 -- 172.21.15.38:0/3920041839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff67c0752b0 msgr2=0x7ff67c199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff67c0752b0 0x7ff67c199d30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00e9f0 tx=0x7ff66c00eeb0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 -- 172.21.15.38:0/3920041839 shutdown_connections 2024-03-26T06:37:34.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6680a2730 0x7ff6680a4bf0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff67c1091f0 0x7ff67c1a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff67c0752b0 0x7ff67c199d30 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 --2- 172.21.15.38:0/3920041839 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff67c073cd0 0x7ff67c1997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.117 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 -- 172.21.15.38:0/3920041839 >> 172.21.15.38:0/3920041839 conn(0x7ff67c0fc8b0 msgr2=0x7ff67c105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:34.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 -- 172.21.15.38:0/3920041839 shutdown_connections 2024-03-26T06:37:34.118 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.114+0000 7ff683875640 1 -- 172.21.15.38:0/3920041839 wait complete. 2024-03-26T06:37:34.127 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-26T06:37:34.233 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.229+0000 7f59a789a640 1 -- 172.21.15.38:0/228209122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a0101dc0 msgr2=0x7f59a0108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.229+0000 7f59a789a640 1 --2- 172.21.15.38:0/228209122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a0101dc0 0x7f59a0108650 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f5998009ca0 tx=0x7f599802c100 comp rx=0 tx=0).stop 2024-03-26T06:37:34.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.229+0000 7f59a789a640 1 -- 172.21.15.38:0/228209122 shutdown_connections 2024-03-26T06:37:34.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.229+0000 7f59a789a640 1 --2- 172.21.15.38:0/228209122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a0101dc0 0x7f59a0108650 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.229+0000 7f59a789a640 1 --2- 172.21.15.38:0/228209122 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59a0101400 0x7f59a0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.229+0000 7f59a789a640 1 --2- 172.21.15.38:0/228209122 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0100210 0x7f59a0100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.229+0000 7f59a789a640 1 -- 172.21.15.38:0/228209122 >> 172.21.15.38:0/228209122 conn(0x7f59a00fb780 msgr2=0x7f59a00fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:34.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.229+0000 7f59a789a640 1 -- 172.21.15.38:0/228209122 shutdown_connections 2024-03-26T06:37:34.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.229+0000 7f59a789a640 1 -- 172.21.15.38:0/228209122 wait complete. 2024-03-26T06:37:34.234 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a789a640 1 Processor -- start 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a789a640 1 -- start start 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a789a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0100210 0x7f59a01041a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a789a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59a0101400 0x7f59a01027e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a789a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a0101dc0 0x7f59a0102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a789a640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a010bea0 con 0x7f59a0101dc0 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a789a640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a010bd20 con 0x7f59a0101400 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a789a640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a010c020 con 0x7f59a0100210 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a5e37640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a0101dc0 0x7f59a0102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a5e37640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a0101dc0 0x7f59a0102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39748/0 (socket says 172.21.15.38:39748) 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a5e37640 1 -- 172.21.15.38:0/1910916162 learned_addr learned my addr 172.21.15.38:0/1910916162 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a4e35640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59a0101400 0x7f59a01027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a5636640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0100210 0x7f59a01041a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:34.235 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a5e37640 1 -- 172.21.15.38:0/1910916162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0100210 msgr2=0x7f59a01041a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a5e37640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0100210 0x7f59a01041a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a5e37640 1 -- 172.21.15.38:0/1910916162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59a0101400 msgr2=0x7f59a01027e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a5e37640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59a0101400 0x7f59a01027e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.230+0000 7f59a5e37640 1 -- 172.21.15.38:0/1910916162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a01035b0 con 0x7f59a0101dc0 2024-03-26T06:37:34.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.231+0000 7f59a5e37640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a0101dc0 0x7f59a0102d20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5998037040 tx=0x7f5998035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:34.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.231+0000 7f59967fc640 1 -- 172.21.15.38:0/1910916162 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998039040 con 0x7f59a0101dc0 2024-03-26T06:37:34.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.231+0000 7f59a789a640 1 -- 172.21.15.38:0/1910916162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a01a15e0 con 0x7f59a0101dc0 2024-03-26T06:37:34.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.231+0000 7f59967fc640 1 -- 172.21.15.38:0/1910916162 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5998044070 con 0x7f59a0101dc0 2024-03-26T06:37:34.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.231+0000 7f59967fc640 1 -- 172.21.15.38:0/1910916162 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59980043a0 con 0x7f59a0101dc0 2024-03-26T06:37:34.236 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.231+0000 7f59a789a640 1 -- 172.21.15.38:0/1910916162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a01a1b80 con 0x7f59a0101dc0 2024-03-26T06:37:34.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.232+0000 7f59a789a640 1 -- 172.21.15.38:0/1910916162 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a01a1ea0 con 0x7f59a0101dc0 2024-03-26T06:37:34.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.233+0000 7f59967fc640 1 -- 172.21.15.38:0/1910916162 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f5998004540 con 0x7f59a0101dc0 2024-03-26T06:37:34.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.233+0000 7f59967fc640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f598c0a26b0 0x7f598c0a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:34.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.234+0000 7f59a5636640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f598c0a26b0 0x7f598c0a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:34.237 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.234+0000 7f59967fc640 1 -- 172.21.15.38:0/1910916162 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f59980f17d0 con 0x7f59a0101dc0 2024-03-26T06:37:34.238 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.234+0000 7f59a5636640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f598c0a26b0 0x7f598c0a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f59a0101260 tx=0x7f599c005f90 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:34.243 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.241+0000 7f59967fc640 1 -- 172.21.15.38:0/1910916162 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f59980beb50 con 0x7f59a0101dc0 2024-03-26T06:37:34.492 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.489+0000 7f59a789a640 1 -- 172.21.15.38:0/1910916162 --> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f59a00fd870 con 0x7f598c0a26b0 2024-03-26T06:37:34.499 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.497+0000 7f59967fc640 1 -- 172.21.15.38:0/1910916162 <== mgr.24883 v2:172.21.15.38:6800/1823987770 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f59a00fd870 con 0x7f598c0a26b0 2024-03-26T06:37:34.499 INFO:teuthology.orchestra.run.smithi038.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-26T00:00:00", "created": "2024-03-26T06:37:33", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 -- 172.21.15.38:0/1910916162 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f598c0a26b0 msgr2=0x7f598c0a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f598c0a26b0 0x7f598c0a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f59a0101260 tx=0x7f599c005f90 comp rx=0 tx=0).stop 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 -- 172.21.15.38:0/1910916162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a0101dc0 msgr2=0x7f59a0102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a0101dc0 0x7f59a0102d20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5998037040 tx=0x7f5998035d90 comp rx=0 tx=0).stop 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 -- 172.21.15.38:0/1910916162 shutdown_connections 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f598c0a26b0 0x7f598c0a4b70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f59a0101dc0 0x7f59a0102d20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59a0101400 0x7f59a01027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 --2- 172.21.15.38:0/1910916162 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f59a0100210 0x7f59a01041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 -- 172.21.15.38:0/1910916162 >> 172.21.15.38:0/1910916162 conn(0x7f59a00fb780 msgr2=0x7f59a00fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 -- 172.21.15.38:0/1910916162 shutdown_connections 2024-03-26T06:37:34.502 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.499+0000 7f59a789a640 1 -- 172.21.15.38:0/1910916162 wait complete. 2024-03-26T06:37:34.513 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-26T06:37:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.621+0000 7f967acea640 1 -- 172.21.15.38:0/3691720612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96741025a0 msgr2=0x7f9674102a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.621+0000 7f967acea640 1 --2- 172.21.15.38:0/3691720612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96741025a0 0x7f9674102a20 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9660009930 tx=0x7f966002bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.621+0000 7f967acea640 1 -- 172.21.15.38:0/3691720612 shutdown_connections 2024-03-26T06:37:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.621+0000 7f967acea640 1 --2- 172.21.15.38:0/3691720612 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9674102f60 0x7f96741097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.621+0000 7f967acea640 1 --2- 172.21.15.38:0/3691720612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f96741025a0 0x7f9674102a20 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.621+0000 7f967acea640 1 --2- 172.21.15.38:0/3691720612 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96740ff5d0 0x7f96740ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.621+0000 7f967acea640 1 -- 172.21.15.38:0/3691720612 >> 172.21.15.38:0/3691720612 conn(0x7f96740faf40 msgr2=0x7f96740fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:34.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.621+0000 7f967acea640 1 -- 172.21.15.38:0/3691720612 shutdown_connections 2024-03-26T06:37:34.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.621+0000 7f967acea640 1 -- 172.21.15.38:0/3691720612 wait complete. 2024-03-26T06:37:34.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.621+0000 7f967acea640 1 Processor -- start 2024-03-26T06:37:34.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.621+0000 7f967acea640 1 -- start start 2024-03-26T06:37:34.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f967acea640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96740ff5d0 0x7f9674199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:34.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f967acea640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96741025a0 0x7f9674199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:34.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f967acea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9674102f60 0x7f96741a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:34.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f967acea640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967410cfc0 con 0x7f9674102f60 2024-03-26T06:37:34.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f967acea640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967410ce40 con 0x7f96740ff5d0 2024-03-26T06:37:34.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f967acea640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967410d140 con 0x7f96741025a0 2024-03-26T06:37:34.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9679287640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9674102f60 0x7f96741a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9673fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96741025a0 0x7f9674199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9678a86640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96740ff5d0 0x7f9674199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9678a86640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96740ff5d0 0x7f9674199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:54352/0 (socket says 172.21.15.38:54352) 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9678a86640 1 -- 172.21.15.38:0/467054686 learned_addr learned my addr 172.21.15.38:0/467054686 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9673fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96741025a0 0x7f9674199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:48716/0 (socket says 172.21.15.38:48716) 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9678a86640 1 -- 172.21.15.38:0/467054686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96741025a0 msgr2=0x7f9674199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9678a86640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96741025a0 0x7f9674199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9678a86640 1 -- 172.21.15.38:0/467054686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9674102f60 msgr2=0x7f96741a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9678a86640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9674102f60 0x7f96741a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9678a86640 1 -- 172.21.15.38:0/467054686 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96741a1400 con 0x7f96740ff5d0 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9678a86640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96740ff5d0 0x7f9674199830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f966800e9c0 tx=0x7f966800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:34.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.622+0000 7f9671ffb640 1 -- 172.21.15.38:0/467054686 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f966800cd80 con 0x7f96740ff5d0 2024-03-26T06:37:34.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.623+0000 7f9671ffb640 1 -- 172.21.15.38:0/467054686 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96680035f0 con 0x7f96740ff5d0 2024-03-26T06:37:34.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.623+0000 7f967acea640 1 -- 172.21.15.38:0/467054686 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96741a16f0 con 0x7f96740ff5d0 2024-03-26T06:37:34.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.623+0000 7f9671ffb640 1 -- 172.21.15.38:0/467054686 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9668010690 con 0x7f96740ff5d0 2024-03-26T06:37:34.629 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.623+0000 7f967acea640 1 -- 172.21.15.38:0/467054686 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96741a1bc0 con 0x7f96740ff5d0 2024-03-26T06:37:34.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.624+0000 7f967acea640 1 -- 172.21.15.38:0/467054686 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96741a1e80 con 0x7f96740ff5d0 2024-03-26T06:37:34.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.625+0000 7f9671ffb640 1 -- 172.21.15.38:0/467054686 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f96680037a0 con 0x7f96740ff5d0 2024-03-26T06:37:34.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.625+0000 7f9671ffb640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f96640a26b0 0x7f96640a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:34.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.625+0000 7f9673fff640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f96640a26b0 0x7f96640a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:34.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.625+0000 7f9671ffb640 1 -- 172.21.15.38:0/467054686 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f96680ceaf0 con 0x7f96740ff5d0 2024-03-26T06:37:34.630 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.626+0000 7f9673fff640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f96640a26b0 0x7f96640a4b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f966002efd0 tx=0x7f9660000bc0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:34.632 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.630+0000 7f9671ffb640 1 -- 172.21.15.38:0/467054686 <== mon.2 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f966809bd50 con 0x7f96740ff5d0 2024-03-26T06:37:34.697 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.693+0000 7f9671ffb640 1 -- 172.21.15.38:0/467054686 <== mon.2 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 23) v1 ==== 137410+0+0 (secure 0 0 0) 0x7f9668097cc0 con 0x7f96740ff5d0 2024-03-26T06:37:34.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:34 smithi038 ceph-mon[32265]: mgrmap e22: x(active, since 1.02126s), standbys: z, y 2024-03-26T06:37:34.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:34 smithi038 ceph-mon[32265]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:34.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:34 smithi038 ceph-mon[32265]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:37:34.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:34 smithi038 ceph-mon[32265]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:34.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:34 smithi038 ceph-mon[32265]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:34.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:34 smithi121 ceph-mon[36349]: mgrmap e22: x(active, since 1.02126s), standbys: z, y 2024-03-26T06:37:34.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:34 smithi121 ceph-mon[36349]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:34.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:34 smithi121 ceph-mon[36349]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:37:34.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:34 smithi121 ceph-mon[36349]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:34.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:34 smithi121 ceph-mon[36349]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:34.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.912+0000 7f967acea640 1 -- 172.21.15.38:0/467054686 --> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9674101c30 con 0x7f96640a26b0 2024-03-26T06:37:34.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:34 smithi170 ceph-mon[36912]: mgrmap e22: x(active, since 1.02126s), standbys: z, y 2024-03-26T06:37:34.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:34 smithi170 ceph-mon[36912]: from='client.24989 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:34.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:34 smithi170 ceph-mon[36912]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:37:34.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:34 smithi170 ceph-mon[36912]: from='client.15366 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:34.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:34 smithi170 ceph-mon[36912]: from='client.15372 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:37:34.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.919+0000 7f9671ffb640 1 -- 172.21.15.38:0/467054686 <== mgr.24883 v2:172.21.15.38:6800/1823987770 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f9674101c30 con 0x7f96640a26b0 2024-03-26T06:37:34.922 INFO:teuthology.orchestra.run.smithi038.stdout:/ 1m 6m3h 2024-03-26T06:37:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.921+0000 7f967acea640 1 -- 172.21.15.38:0/467054686 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f96640a26b0 msgr2=0x7f96640a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.921+0000 7f967acea640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f96640a26b0 0x7f96640a4b70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f966002efd0 tx=0x7f9660000bc0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.921+0000 7f967acea640 1 -- 172.21.15.38:0/467054686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96740ff5d0 msgr2=0x7f9674199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.921+0000 7f967acea640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96740ff5d0 0x7f9674199830 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f966800e9c0 tx=0x7f966800ee80 comp rx=0 tx=0).stop 2024-03-26T06:37:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.921+0000 7f967acea640 1 -- 172.21.15.38:0/467054686 shutdown_connections 2024-03-26T06:37:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.921+0000 7f967acea640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f96640a26b0 0x7f96640a4b70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.921+0000 7f967acea640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9674102f60 0x7f96741a0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.921+0000 7f967acea640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f96741025a0 0x7f9674199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.921+0000 7f967acea640 1 --2- 172.21.15.38:0/467054686 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f96740ff5d0 0x7f9674199830 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.922+0000 7f967acea640 1 -- 172.21.15.38:0/467054686 >> 172.21.15.38:0/467054686 conn(0x7f96740faf40 msgr2=0x7f96740fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.922+0000 7f967acea640 1 -- 172.21.15.38:0/467054686 shutdown_connections 2024-03-26T06:37:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:37:34.922+0000 7f967acea640 1 -- 172.21.15.38:0/467054686 wait complete. 2024-03-26T06:37:34.940 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-26T06:37:34.969 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-26T06:37:34.981 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-26T06:37:35.039 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T06:37:35.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-26T06:37:35.042 INFO:teuthology.run_tasks:Running task workunit... 2024-03-26T06:37:35.075 INFO:tasks.workunit:Pulling workunits from ref 7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:37:35.075 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-26T06:37:35.075 INFO:tasks.workunit:timeout=3h 2024-03-26T06:37:35.076 INFO:tasks.workunit:cleanup=True 2024-03-26T06:37:35.076 DEBUG:teuthology.orchestra.run.smithi038:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-26T06:37:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-26T06:37:35.091 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-26T06:37:35.091 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-26T06:37:35.091 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T06:37:35.091 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-26T06:37:35.091 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2024-03-26 06:35:33.590708878 +0000 2024-03-26T06:37:35.091 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2024-03-26 06:35:33.590708878 +0000 2024-03-26T06:37:35.092 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2024-03-26 06:37:23.104049620 +0000 2024-03-26T06:37:35.092 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: 2024-03-26 06:35:33.590708878 +0000 2024-03-26T06:37:35.092 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-26T06:37:35.092 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-26T06:37:35.169 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7e98f5f6b535defc7fa715b456904ffc0500aa9f 2024-03-26T06:37:35.224 INFO:tasks.workunit.client.0.smithi038.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-26T06:37:36.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:35 smithi038 ceph-mon[32265]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-03-26T06:37:36.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:35 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:35 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:35 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:35 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:35 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:35 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:35 smithi038 ceph-mon[32265]: [26/Mar/2024:06:37:34] ENGINE Bus STARTING 2024-03-26T06:37:36.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:35 smithi038 ceph-mon[32265]: [26/Mar/2024:06:37:35] ENGINE Serving on http://172.21.15.38:8765 2024-03-26T06:37:36.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:35 smithi038 ceph-mon[32265]: [26/Mar/2024:06:37:35] ENGINE Serving on https://172.21.15.38:7150 2024-03-26T06:37:36.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:35 smithi038 ceph-mon[32265]: [26/Mar/2024:06:37:35] ENGINE Bus STARTED 2024-03-26T06:37:36.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:35 smithi121 ceph-mon[36349]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-03-26T06:37:36.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:35 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:35 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:35 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:35 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:35 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:35 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:35 smithi121 ceph-mon[36349]: [26/Mar/2024:06:37:34] ENGINE Bus STARTING 2024-03-26T06:37:36.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:35 smithi121 ceph-mon[36349]: [26/Mar/2024:06:37:35] ENGINE Serving on http://172.21.15.38:8765 2024-03-26T06:37:36.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:35 smithi121 ceph-mon[36349]: [26/Mar/2024:06:37:35] ENGINE Serving on https://172.21.15.38:7150 2024-03-26T06:37:36.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:35 smithi121 ceph-mon[36349]: [26/Mar/2024:06:37:35] ENGINE Bus STARTED 2024-03-26T06:37:36.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:35 smithi170 ceph-mon[36912]: mgrmap e23: x(active, since 2s), standbys: z, y 2024-03-26T06:37:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:35 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:35 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:35 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:35 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:35 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:35 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:35 smithi170 ceph-mon[36912]: [26/Mar/2024:06:37:34] ENGINE Bus STARTING 2024-03-26T06:37:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:35 smithi170 ceph-mon[36912]: [26/Mar/2024:06:37:35] ENGINE Serving on http://172.21.15.38:8765 2024-03-26T06:37:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:35 smithi170 ceph-mon[36912]: [26/Mar/2024:06:37:35] ENGINE Serving on https://172.21.15.38:7150 2024-03-26T06:37:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:35 smithi170 ceph-mon[36912]: [26/Mar/2024:06:37:35] ENGINE Bus STARTED 2024-03-26T06:37:37.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Unable to set osd_memory_target on smithi170 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi121 to 174.1M 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Unable to set osd_memory_target on smithi121 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Unable to set osd_memory_target on smithi038 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Updating smithi121:/etc/ceph/ceph.conf 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Updating smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Updating smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Updating smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:37:37.336 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:36 smithi038 ceph-mon[32265]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:37:37.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.337 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Unable to set osd_memory_target on smithi170 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi121 to 174.1M 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Unable to set osd_memory_target on smithi121 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.338 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Unable to set osd_memory_target on smithi038 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Updating smithi121:/etc/ceph/ceph.conf 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Updating smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Updating smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Updating smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:37:37.339 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:36 smithi170 ceph-mon[36912]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:37:37.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Unable to set osd_memory_target on smithi170 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi121 to 174.1M 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Unable to set osd_memory_target on smithi121 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Unable to set osd_memory_target on smithi038 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:37:37.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Updating smithi038:/etc/ceph/ceph.conf 2024-03-26T06:37:37.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Updating smithi121:/etc/ceph/ceph.conf 2024-03-26T06:37:37.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Updating smithi170:/etc/ceph/ceph.conf 2024-03-26T06:37:37.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Updating smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:37:37.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Updating smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:37:37.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Updating smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T06:37:37.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Updating smithi121:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:37:37.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:37:37.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:36 smithi121 ceph-mon[36349]: Updating smithi170:/etc/ceph/ceph.client.admin.keyring 2024-03-26T06:37:38.270 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:37:38.270 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: Updating smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:37:38.270 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: Updating smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:37:38.270 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: Updating smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:37:38.271 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:37:38.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: Updating smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:37:38.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: Updating smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:37:38.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: Updating smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:37:38.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-26T06:37:38.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:37:38.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:37:38.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:37:38.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:37:38.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: Updating smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:37:38.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: Updating smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:37:38.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: Updating smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:37:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:37:40.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:40 smithi038 ceph-mon[32265]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:37:40.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:40 smithi121 ceph-mon[36349]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:37:40.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:40 smithi170 ceph-mon[36912]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-26T06:37:42.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:42 smithi038 ceph-mon[32265]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-26T06:37:42.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:42 smithi121 ceph-mon[36349]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-26T06:37:42.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:42 smithi170 ceph-mon[36912]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-26T06:37:44.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:44 smithi038 ceph-mon[32265]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-26T06:37:44.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:44 smithi121 ceph-mon[36349]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-26T06:37:44.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:44 smithi170 ceph-mon[36912]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 59 op/s 2024-03-26T06:37:46.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:46 smithi038 ceph-mon[32265]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-26T06:37:46.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:46 smithi121 ceph-mon[36349]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-26T06:37:46.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:46 smithi170 ceph-mon[36912]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-26T06:37:48.192 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.187+0000 7fb793812640 1 -- 172.21.15.38:0/3222198396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c102530 msgr2=0x7fb78c102910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.187+0000 7fb793812640 1 --2- 172.21.15.38:0/3222198396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c102530 0x7fb78c102910 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb788009900 tx=0x7fb78802bc30 comp rx=0 tx=0).stop 2024-03-26T06:37:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.187+0000 7fb793812640 1 -- 172.21.15.38:0/3222198396 shutdown_connections 2024-03-26T06:37:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.187+0000 7fb793812640 1 --2- 172.21.15.38:0/3222198396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c10b900 0x7fb78c10dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.187+0000 7fb793812640 1 --2- 172.21.15.38:0/3222198396 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb78c102e50 0x7fb78c10b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.187+0000 7fb793812640 1 --2- 172.21.15.38:0/3222198396 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c102530 0x7fb78c102910 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.188+0000 7fb793812640 1 -- 172.21.15.38:0/3222198396 >> 172.21.15.38:0/3222198396 conn(0x7fb78c077320 msgr2=0x7fb78c077730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.188+0000 7fb793812640 1 -- 172.21.15.38:0/3222198396 shutdown_connections 2024-03-26T06:37:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.188+0000 7fb793812640 1 -- 172.21.15.38:0/3222198396 wait complete. 2024-03-26T06:37:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.188+0000 7fb793812640 1 Processor -- start 2024-03-26T06:37:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.188+0000 7fb793812640 1 -- start start 2024-03-26T06:37:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.188+0000 7fb793812640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c102530 0x7fb78c100180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.188+0000 7fb793812640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb78c102e50 0x7fb78c1006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.188+0000 7fb793812640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c10b900 0x7fb78c100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.188+0000 7fb793812640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c111710 con 0x7fb78c102530 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb793812640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c111590 con 0x7fb78c10b900 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb793812640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c111890 con 0x7fb78c102e50 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7915ae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c102530 0x7fb78c100180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7915ae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c102530 0x7fb78c100180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42112/0 (socket says 172.21.15.38:42112) 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7915ae640 1 -- 172.21.15.38:0/1448882783 learned_addr learned my addr 172.21.15.38:0/1448882783 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb790dad640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb78c102e50 0x7fb78c1006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb791daf640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c10b900 0x7fb78c100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7915ae640 1 -- 172.21.15.38:0/1448882783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb78c102e50 msgr2=0x7fb78c1006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7915ae640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb78c102e50 0x7fb78c1006c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7915ae640 1 -- 172.21.15.38:0/1448882783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c10b900 msgr2=0x7fb78c100c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7915ae640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c10b900 0x7fb78c100c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7915ae640 1 -- 172.21.15.38:0/1448882783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c1a5ef0 con 0x7fb78c102530 2024-03-26T06:37:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7915ae640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c102530 0x7fb78c100180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb78802c240 tx=0x7fb7880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7827fc640 1 -- 172.21.15.38:0/1448882783 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788043020 con 0x7fb78c102530 2024-03-26T06:37:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7827fc640 1 -- 172.21.15.38:0/1448882783 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7880054e0 con 0x7fb78c102530 2024-03-26T06:37:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb793812640 1 -- 172.21.15.38:0/1448882783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c1a6180 con 0x7fb78c102530 2024-03-26T06:37:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb7827fc640 1 -- 172.21.15.38:0/1448882783 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb788003b50 con 0x7fb78c102530 2024-03-26T06:37:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.189+0000 7fb793812640 1 -- 172.21.15.38:0/1448882783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c1a6720 con 0x7fb78c102530 2024-03-26T06:37:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.192+0000 7fb7827fc640 1 -- 172.21.15.38:0/1448882783 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) v1 ==== 137466+0+0 (secure 0 0 0) 0x7fb788046020 con 0x7fb78c102530 2024-03-26T06:37:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.192+0000 7fb7963f2740 1 Processor -- start 2024-03-26T06:37:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.192+0000 7fb7827fc640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb7780a2b30 0x7fb7780a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.192+0000 7fb7963f2740 1 -- start start 2024-03-26T06:37:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb7827fc640 1 -- 172.21.15.38:0/1448882783 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fb7880f1020 con 0x7fb78c102530 2024-03-26T06:37:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb791daf640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb7780a2b30 0x7fb7780a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2bd40 0x55b292b2aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b2b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b34550 0x55b292b36940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb7963f2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b292a8f5f0 con 0x55b292b2afd0 2024-03-26T06:37:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb7963f2740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b292a38930 con 0x55b292b2bd40 2024-03-26T06:37:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb7963f2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b292a8b690 con 0x55b292b34550 2024-03-26T06:37:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb790dad640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b2b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb790dad640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b2b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42114/0 (socket says 172.21.15.38:42114) 2024-03-26T06:37:48.203 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb790dad640 1 -- 172.21.15.38:0/2924630908 learned_addr learned my addr 172.21.15.38:0/2924630908 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb7915ae640 1 --2- 172.21.15.38:0/2924630908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2bd40 0x55b292b2aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb791daf640 1 --2- 172.21.15.38:0/2924630908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b34550 0x55b292b36940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb790dad640 1 -- 172.21.15.38:0/2924630908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b34550 msgr2=0x55b292b36940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb790dad640 1 --2- 172.21.15.38:0/2924630908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b34550 0x55b292b36940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb791daf640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb7780a2b30 0x7fb7780a4ff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb784005fd0 tx=0x7fb7840107a0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb790dad640 1 -- 172.21.15.38:0/2924630908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2bd40 msgr2=0x55b292b2aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb790dad640 1 --2- 172.21.15.38:0/2924630908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2bd40 0x55b292b2aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb790dad640 1 -- 172.21.15.38:0/2924630908 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b2927a4210 con 0x55b292b2afd0 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.193+0000 7fb790dad640 1 --2- 172.21.15.38:0/2924630908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b2b450 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00cc40 tx=0x7fb77c00e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84160a6b760c00 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7777fe640 1 -- 172.21.15.38:0/2924630908 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c014070 con 0x55b292b2afd0 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7777fe640 1 -- 172.21.15.38:0/2924630908 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb77c0047a0 con 0x55b292b2afd0 2024-03-26T06:37:48.204 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7777fe640 1 -- 172.21.15.38:0/2924630908 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb77c010760 con 0x55b292b2afd0 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7963f2740 1 -- 172.21.15.38:0/2924630908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 msgr2=0x55b292b2b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7963f2740 1 --2- 172.21.15.38:0/2924630908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b2b450 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fb77c00cc40 tx=0x7fb77c00e720 comp rx=0 tx=0).stop 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7963f2740 1 -- 172.21.15.38:0/2924630908 shutdown_connections 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7963f2740 1 --2- 172.21.15.38:0/2924630908 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b34550 0x55b292b36940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7963f2740 1 --2- 172.21.15.38:0/2924630908 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b2b450 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7963f2740 1 --2- 172.21.15.38:0/2924630908 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2bd40 0x55b292b2aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7963f2740 1 -- 172.21.15.38:0/2924630908 >> 172.21.15.38:0/2924630908 conn(0x55b292af25b0 msgr2=0x55b292b25cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7963f2740 1 -- 172.21.15.38:0/2924630908 shutdown_connections 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.194+0000 7fb7963f2740 1 -- 172.21.15.38:0/2924630908 wait complete. 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7963f2740 1 -- start start 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7963f2740 1 Processor -- start 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b315b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2bd40 0x55b292b31af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b34550 0x55b292b32030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7963f2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b292b37280 con 0x55b292b2afd0 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7963f2740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b292a4bd90 con 0x55b292b2bd40 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7963f2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b292b37400 con 0x55b292b34550 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7963f2740 10 client.0 _ll_register_callbacks cb 0x55b2927a3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7915ae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b315b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7915ae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b315b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42118/0 (socket says 172.21.15.38:42118) 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.195+0000 7fb7915ae640 1 -- 172.21.15.38:0/1113408635 learned_addr learned my addr 172.21.15.38:0/1113408635 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb791daf640 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b34550 0x55b292b32030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb790dad640 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2bd40 0x55b292b31af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb7915ae640 1 -- 172.21.15.38:0/1113408635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b34550 msgr2=0x55b292b32030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb7915ae640 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b34550 0x55b292b32030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb7915ae640 1 -- 172.21.15.38:0/1113408635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2bd40 msgr2=0x55b292b31af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb7915ae640 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2bd40 0x55b292b31af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb7915ae640 1 -- 172.21.15.38:0/1113408635 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b292870620 con 0x55b292b2afd0 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb790dad640 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2bd40 0x55b292b31af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb7915ae640 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b315b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb7880bd5a0 tx=0x7fb7880035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87732e3d17f6eb1d server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb774ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb774ff9640 1 -- 172.21.15.38:0/1113408635 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7880bdd10 con 0x55b292b2afd0 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb774ff9640 1 -- 172.21.15.38:0/1113408635 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fb7880ee950 con 0x55b292b2afd0 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb7963f2740 1 -- 172.21.15.38:0/1113408635 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb77c0047a0 con 0x55b292b2afd0 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb774ff9640 1 -- 172.21.15.38:0/1113408635 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7880eeb50 con 0x55b292b2afd0 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb7963f2740 10 client.15396 fetch_fsmap learned FSMap version 23 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb7963f2740 1 -- 172.21.15.38:0/1113408635 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b292b435c0 con 0x55b292b2afd0 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.196+0000 7fb774ff9640 1 -- 172.21.15.38:0/1113408635 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb77c0047a0 con 0x55b292b2afd0 2024-03-26T06:37:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.197+0000 7fb774ff9640 1 -- 172.21.15.38:0/1113408635 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb7880bd870 con 0x55b292b2afd0 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.197+0000 7fb7963f2740 10 client.15396 fetch_fsmap finished waiting for FSMap version 23 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.197+0000 7fb7963f2740 10 client.15396 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.197+0000 7fb7963f2740 20 client.15396 populate_metadata read hostname 'smithi038' 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.197+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55b292b44a20 0x55b292b46e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.197+0000 7fb7963f2740 4 client.15396 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.197+0000 7fb7963f2740 1 -- 172.21.15.38:0/1113408635 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b292a9cb30 con 0x55b292b44a20 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.197+0000 7fb790dad640 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55b292b44a20 0x55b292b46e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.197+0000 7fb774ff9640 10 client.15396 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.197+0000 7fb790dad640 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55b292b44a20 0x55b292b46e10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac56aa3aaa2be260 server_cookie=7cee7ca9a6bccf89 in_seq=0 out_seq=0 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb774ff9640 1 -- 172.21.15.38:0/1113408635 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b292a9cb30 con 0x55b292b44a20 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb774ff9640 10 client.15396 handle_command_reply: tid=0 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 1 client.15396 shutdown 2024-03-26T06:37:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 1 -- 172.21.15.38:0/1113408635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 msgr2=0x55b292b315b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b315b0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fb7880bd5a0 tx=0x7fb7880035f0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 1 -- 172.21.15.38:0/1113408635 shutdown_connections 2024-03-26T06:37:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55b292b44a20 0x55b292b46e10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b34550 0x55b292b32030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2bd40 0x55b292b31af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb774ff9640 0 client.15396 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:37:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1113408635 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b315b0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 1 -- 172.21.15.38:0/1113408635 >> 172.21.15.38:0/1113408635 conn(0x55b292af25b0 msgr2=0x55b292b27a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 1 -- 172.21.15.38:0/1113408635 shutdown_connections 2024-03-26T06:37:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 1 -- 172.21.15.38:0/1113408635 wait complete. 2024-03-26T06:37:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.198+0000 7fb7963f2740 20 client.15396 trim_cache size 0 max 16384 2024-03-26T06:37:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb7963f2740 1 Processor -- start 2024-03-26T06:37:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb7963f2740 1 -- start start 2024-03-26T06:37:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b33ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.219 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b2bd40 0x55b292b3b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b34550 0x55b292b3bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb7963f2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1b13a0 con 0x55b292b2afd0 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb7963f2740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1aef40 con 0x55b292b34550 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb7963f2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1acfe0 con 0x55b292b2bd40 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb7915ae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b33ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb7915ae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b33ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42126/0 (socket says 172.21.15.38:42126) 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb7915ae640 1 -- 172.21.15.38:0/107003348 learned_addr learned my addr 172.21.15.38:0/107003348 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb791daf640 1 --2- 172.21.15.38:0/107003348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b34550 0x55b292b3bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.212+0000 7fb790dad640 1 --2- 172.21.15.38:0/107003348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b2bd40 0x55b292b3b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb7915ae640 1 -- 172.21.15.38:0/107003348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b2bd40 msgr2=0x55b292b3b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb7915ae640 1 --2- 172.21.15.38:0/107003348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b2bd40 0x55b292b3b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb7915ae640 1 -- 172.21.15.38:0/107003348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b34550 msgr2=0x55b292b3bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb7915ae640 1 --2- 172.21.15.38:0/107003348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b34550 0x55b292b3bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.220 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb7915ae640 1 -- 172.21.15.38:0/107003348 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b292b2a2a0 con 0x55b292b2afd0 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb790dad640 1 --2- 172.21.15.38:0/107003348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b2bd40 0x55b292b3b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb7915ae640 1 --2- 172.21.15.38:0/107003348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b33ce0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb788000c00 tx=0x7fb7880b99e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88a826718c720d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb776ffd640 1 -- 172.21.15.38:0/107003348 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7880bd3e0 con 0x55b292b2afd0 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb776ffd640 1 -- 172.21.15.38:0/107003348 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7880b9040 con 0x55b292b2afd0 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb776ffd640 1 -- 172.21.15.38:0/107003348 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7880bd150 con 0x55b292b2afd0 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb7963f2740 1 -- 172.21.15.38:0/107003348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 msgr2=0x55b292b33ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.213+0000 7fb7963f2740 1 --2- 172.21.15.38:0/107003348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b33ce0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb788000c00 tx=0x7fb7880b99e0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 -- 172.21.15.38:0/107003348 shutdown_connections 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 --2- 172.21.15.38:0/107003348 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b34550 0x55b292b3bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 --2- 172.21.15.38:0/107003348 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b2bd40 0x55b292b3b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 --2- 172.21.15.38:0/107003348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b2afd0 0x55b292b33ce0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 -- 172.21.15.38:0/107003348 >> 172.21.15.38:0/107003348 conn(0x55b292af25b0 msgr2=0x55b292b36710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 -- 172.21.15.38:0/107003348 shutdown_connections 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 -- 172.21.15.38:0/107003348 wait complete. 2024-03-26T06:37:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 -- start start 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 Processor -- start 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2afd0 0x55b292b37950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b2bd40 0x55b292b37e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b34550 0x55b292b93f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1ad5e0 con 0x55b292b34550 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1ae8e0 con 0x55b292b2afd0 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c1adc20 con 0x55b292b2bd40 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb7963f2740 10 client.0 _ll_register_callbacks cb 0x7fb77c00c3d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb791daf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b34550 0x55b292b93f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb791daf640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b34550 0x55b292b93f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42140/0 (socket says 172.21.15.38:42140) 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.214+0000 7fb791daf640 1 -- 172.21.15.38:0/1043423530 learned_addr learned my addr 172.21.15.38:0/1043423530 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:37:48.222 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb790dad640 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b2bd40 0x55b292b37e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb7915ae640 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2afd0 0x55b292b37950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb791daf640 1 -- 172.21.15.38:0/1043423530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b2bd40 msgr2=0x55b292b37e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb791daf640 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b2bd40 0x55b292b37e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb791daf640 1 -- 172.21.15.38:0/1043423530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2afd0 msgr2=0x55b292b37950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb791daf640 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2afd0 0x55b292b37950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb791daf640 1 -- 172.21.15.38:0/1043423530 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b292b945a0 con 0x55b292b34550 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb7915ae640 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2afd0 0x55b292b37950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb791daf640 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b34550 0x55b292b93f30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb7840139f0 tx=0x7fb784015070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aa6446a4912a346 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb774ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb774ff9640 1 -- 172.21.15.38:0/1043423530 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784003b70 con 0x55b292b34550 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb774ff9640 1 -- 172.21.15.38:0/1043423530 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fb784003d10 con 0x55b292b34550 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb7963f2740 1 -- 172.21.15.38:0/1043423530 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7880b9040 con 0x55b292b34550 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb774ff9640 1 -- 172.21.15.38:0/1043423530 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb784004050 con 0x55b292b34550 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb7963f2740 10 client.15402 fetch_fsmap learned FSMap version 23 2024-03-26T06:37:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.215+0000 7fb7963f2740 1 -- 172.21.15.38:0/1043423530 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55b292b9b920 con 0x55b292b34550 2024-03-26T06:37:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.216+0000 7fb774ff9640 1 -- 172.21.15.38:0/1043423530 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7880b9040 con 0x55b292b34550 2024-03-26T06:37:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.216+0000 7fb774ff9640 1 -- 172.21.15.38:0/1043423530 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb784028070 con 0x55b292b34550 2024-03-26T06:37:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.216+0000 7fb7963f2740 10 client.15402 fetch_fsmap finished waiting for FSMap version 23 2024-03-26T06:37:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.216+0000 7fb7963f2740 10 client.15402 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:37:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.216+0000 7fb7963f2740 20 client.15402 populate_metadata read hostname 'smithi038' 2024-03-26T06:37:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.216+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55b292b9c0d0 0x55b292b9e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:37:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.216+0000 7fb7963f2740 4 client.15402 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:37:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.216+0000 7fb7963f2740 1 -- 172.21.15.38:0/1043423530 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b292a96e80 con 0x55b292b9c0d0 2024-03-26T06:37:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.216+0000 7fb7915ae640 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55b292b9c0d0 0x55b292b9e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:37:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.217+0000 7fb774ff9640 10 client.15402 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:37:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.217+0000 7fb7915ae640 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55b292b9c0d0 0x55b292b9e4f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46df2c203018708b server_cookie=9dce57b7b48af09a in_seq=0 out_seq=0 2024-03-26T06:37:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb774ff9640 1 -- 172.21.15.38:0/1043423530 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b292a96e80 con 0x55b292b9c0d0 2024-03-26T06:37:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb774ff9640 10 client.15402 handle_command_reply: tid=0 2024-03-26T06:37:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 1 client.15402 shutdown 2024-03-26T06:37:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 1 -- 172.21.15.38:0/1043423530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b34550 msgr2=0x55b292b93f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b34550 0x55b292b93f30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fb7840139f0 tx=0x7fb784015070 comp rx=0 tx=0).stop 2024-03-26T06:37:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 1 -- 172.21.15.38:0/1043423530 shutdown_connections 2024-03-26T06:37:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55b292b9c0d0 0x55b292b9e4f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b292b34550 0x55b292b93f30 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55b292b2bd40 0x55b292b37e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb774ff9640 0 client.15402 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:37:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 1 --2- 172.21.15.38:0/1043423530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55b292b2afd0 0x55b292b37950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 1 -- 172.21.15.38:0/1043423530 >> 172.21.15.38:0/1043423530 conn(0x55b292af25b0 msgr2=0x55b292b2fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 1 -- 172.21.15.38:0/1043423530 shutdown_connections 2024-03-26T06:37:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 1 -- 172.21.15.38:0/1043423530 wait complete. 2024-03-26T06:37:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.229+0000 7fb7963f2740 20 client.15402 trim_cache size 0 max 16384 2024-03-26T06:37:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:37:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:37:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:37:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:37:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 -- 172.21.15.38:0/1448882783 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb7780a2b30 msgr2=0x7fb7780a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb7780a2b30 0x7fb7780a4ff0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb784005fd0 tx=0x7fb7840107a0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 -- 172.21.15.38:0/1448882783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c102530 msgr2=0x7fb78c100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:37:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c102530 0x7fb78c100180 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fb78802c240 tx=0x7fb7880042f0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 -- 172.21.15.38:0/1448882783 shutdown_connections 2024-03-26T06:37:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb7780a2b30 0x7fb7780a4ff0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb78c10b900 0x7fb78c100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb78c102e50 0x7fb78c1006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 --2- 172.21.15.38:0/1448882783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb78c102530 0x7fb78c100180 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:37:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 -- 172.21.15.38:0/1448882783 >> 172.21.15.38:0/1448882783 conn(0x7fb78c077320 msgr2=0x7fb78c0fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:37:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 -- 172.21.15.38:0/1448882783 shutdown_connections 2024-03-26T06:37:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:37:48.230+0000 7fb793812640 1 -- 172.21.15.38:0/1448882783 wait complete. 2024-03-26T06:37:48.323 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:48 smithi170 ceph-mon[36912]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-26T06:37:48.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:48 smithi038 ceph-mon[32265]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-26T06:37:48.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:48 smithi121 ceph-mon[36349]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-26T06:37:50.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:50 smithi038 ceph-mon[32265]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-26T06:37:50.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:50 smithi121 ceph-mon[36349]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-26T06:37:50.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:50 smithi170 ceph-mon[36912]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-26T06:37:52.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:52 smithi038 ceph-mon[32265]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 79 KiB/s wr, 54 op/s 2024-03-26T06:37:52.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:52 smithi121 ceph-mon[36349]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 79 KiB/s wr, 54 op/s 2024-03-26T06:37:52.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:52 smithi170 ceph-mon[36912]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 79 KiB/s wr, 54 op/s 2024-03-26T06:37:54.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:54 smithi038 ceph-mon[32265]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-26T06:37:54.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:54 smithi121 ceph-mon[36349]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-26T06:37:54.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:54 smithi170 ceph-mon[36912]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-26T06:37:56.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:56 smithi038 ceph-mon[32265]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-26T06:37:56.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:56 smithi121 ceph-mon[36349]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-26T06:37:56.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:56 smithi170 ceph-mon[36912]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-26T06:37:58.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:37:58 smithi038 ceph-mon[32265]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-26T06:37:58.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:37:58 smithi121 ceph-mon[36349]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-26T06:37:58.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:37:58 smithi170 ceph-mon[36912]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-26T06:38:00.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:00 smithi038 ceph-mon[32265]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-26T06:38:00.334 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:38:00 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:38:00.124+0000 7f646a1d4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:38:00.334 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:38:00 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:38:00.124+0000 7f646a1d4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:38:00.334 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:38:00 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:38:00.124+0000 7f646a1d4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:38:00.334 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:38:00 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:38:00.124+0000 7f646a1d4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:38:00.334 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:38:00 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:38:00.124+0000 7f646a1d4700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-26T06:38:00.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:00 smithi121 ceph-mon[36349]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-26T06:38:00.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:00 smithi170 ceph-mon[36912]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-03-26T06:38:01.331 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.328+0000 7f7d02553640 1 -- 172.21.15.38:0/2756872351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cfc105950 msgr2=0x7f7cfc105d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.328+0000 7f7d02553640 1 --2- 172.21.15.38:0/2756872351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cfc105950 0x7f7cfc105d30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009900 tx=0x7f7cec02bc30 comp rx=0 tx=0).stop 2024-03-26T06:38:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.328+0000 7f7d02553640 1 -- 172.21.15.38:0/2756872351 shutdown_connections 2024-03-26T06:38:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.328+0000 7f7d02553640 1 --2- 172.21.15.38:0/2756872351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc100300 0x7f7cfc10c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.328+0000 7f7d02553640 1 --2- 172.21.15.38:0/2756872351 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cfc0ff940 0x7f7cfc0ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.328+0000 7f7d02553640 1 --2- 172.21.15.38:0/2756872351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cfc105950 0x7f7cfc105d30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.328+0000 7f7d02553640 1 -- 172.21.15.38:0/2756872351 >> 172.21.15.38:0/2756872351 conn(0x7f7cfc0767b0 msgr2=0x7f7cfc0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.328+0000 7f7d02553640 1 -- 172.21.15.38:0/2756872351 shutdown_connections 2024-03-26T06:38:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.328+0000 7f7d02553640 1 -- 172.21.15.38:0/2756872351 wait complete. 2024-03-26T06:38:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.329+0000 7f7d02553640 1 Processor -- start 2024-03-26T06:38:01.332 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.329+0000 7f7d02553640 1 -- start start 2024-03-26T06:38:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d02553640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cfc0ff940 0x7f7cfc19d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d02553640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc100300 0x7f7cfc19dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d02553640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cfc105950 0x7f7cfc1978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d02553640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc111770 con 0x7f7cfc105950 2024-03-26T06:38:01.334 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d02553640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1115f0 con 0x7f7cfc100300 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d02553640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1118f0 con 0x7f7cfc0ff940 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d00af0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cfc105950 0x7f7cfc1978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d00af0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cfc105950 0x7f7cfc1978b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53356/0 (socket says 172.21.15.38:53356) 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d00af0640 1 -- 172.21.15.38:0/3637523555 learned_addr learned my addr 172.21.15.38:0/3637523555 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cfc0ff940 0x7f7cfc19d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc100300 0x7f7cfc19dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d00af0640 1 -- 172.21.15.38:0/3637523555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cfc0ff940 msgr2=0x7f7cfc19d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d00af0640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cfc0ff940 0x7f7cfc19d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d00af0640 1 -- 172.21.15.38:0/3637523555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc100300 msgr2=0x7f7cfc19dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d00af0640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc100300 0x7f7cfc19dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d00af0640 1 -- 172.21.15.38:0/3637523555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc198030 con 0x7f7cfc105950 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d00af0640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cfc105950 0x7f7cfc1978b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7cf400eb10 tx=0x7f7cf400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:01.335 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7cf97fa640 1 -- 172.21.15.38:0/3637523555 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4004b20 con 0x7f7cfc105950 2024-03-26T06:38:01.336 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7d02553640 1 -- 172.21.15.38:0/3637523555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cfc198320 con 0x7f7cfc105950 2024-03-26T06:38:01.336 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7cf97fa640 1 -- 172.21.15.38:0/3637523555 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cf4004cc0 con 0x7f7cfc105950 2024-03-26T06:38:01.336 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.330+0000 7f7cf97fa640 1 -- 172.21.15.38:0/3637523555 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4010760 con 0x7f7cfc105950 2024-03-26T06:38:01.336 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.331+0000 7f7d02553640 1 -- 172.21.15.38:0/3637523555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cfc1a45c0 con 0x7f7cfc105950 2024-03-26T06:38:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7cf97fa640 1 -- 172.21.15.38:0/3637523555 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f7cf4005020 con 0x7f7cfc105950 2024-03-26T06:38:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7d05212740 1 Processor -- start 2024-03-26T06:38:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7d05212740 1 -- start start 2024-03-26T06:38:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7d05212740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a94d40 0x5579c3a93a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7d05212740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a93fd0 0x5579c3a94450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7d05212740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 0x5579c3a9f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7d05212740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579c39f85f0 con 0x5579c3a93fd0 2024-03-26T06:38:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7d05212740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579c39a1930 con 0x5579c3a94d40 2024-03-26T06:38:01.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7d05212740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579c39f4690 con 0x5579c3a9d550 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7cfbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a93fd0 0x5579c3a94450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7cf97fa640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f7ce80a2b30 0x7f7ce80a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7cfbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a93fd0 0x5579c3a94450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53366/0 (socket says 172.21.15.38:53366) 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.333+0000 7f7cfbfff640 1 -- 172.21.15.38:0/1880805606 learned_addr learned my addr 172.21.15.38:0/1880805606 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/1880805606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 0x5579c3a9f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7d00af0640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f7ce80a2b30 0x7f7ce80a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/1880805606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a94d40 0x5579c3a93a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cf97fa640 1 -- 172.21.15.38:0/3637523555 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f7cf40cf6f0 con 0x7f7cfc105950 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cfbfff640 1 -- 172.21.15.38:0/1880805606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 msgr2=0x5579c3a9f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/1880805606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 0x5579c3a9f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cfbfff640 1 -- 172.21.15.38:0/1880805606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a94d40 msgr2=0x5579c3a93a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/1880805606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a94d40 0x5579c3a93a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cfbfff640 1 -- 172.21.15.38:0/1880805606 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579c370d210 con 0x5579c3a93fd0 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/1880805606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 0x5579c3a9f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/1880805606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a93fd0 0x5579c3a94450 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009900 tx=0x7f7cec005be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715f013363156fec server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7d00af0640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f7ce80a2b30 0x7f7ce80a4ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40ccd10 tx=0x7f7cf4090f80 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cf27fc640 1 -- 172.21.15.38:0/1880805606 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec045070 con 0x5579c3a93fd0 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cf27fc640 1 -- 172.21.15.38:0/1880805606 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cec02c880 con 0x5579c3a93fd0 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.334+0000 7f7cf27fc640 1 -- 172.21.15.38:0/1880805606 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec004ec0 con 0x5579c3a93fd0 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.335+0000 7f7d05212740 1 -- 172.21.15.38:0/1880805606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a93fd0 msgr2=0x5579c3a94450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.335+0000 7f7d05212740 1 --2- 172.21.15.38:0/1880805606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a93fd0 0x5579c3a94450 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009900 tx=0x7f7cec005be0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.335+0000 7f7d05212740 1 -- 172.21.15.38:0/1880805606 shutdown_connections 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.335+0000 7f7d05212740 1 --2- 172.21.15.38:0/1880805606 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 0x5579c3a9f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.335+0000 7f7d05212740 1 --2- 172.21.15.38:0/1880805606 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a93fd0 0x5579c3a94450 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.335+0000 7f7d05212740 1 --2- 172.21.15.38:0/1880805606 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a94d40 0x5579c3a93a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.335+0000 7f7d05212740 1 -- 172.21.15.38:0/1880805606 >> 172.21.15.38:0/1880805606 conn(0x5579c3a5b5b0 msgr2=0x5579c3a8ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.335+0000 7f7d05212740 1 -- 172.21.15.38:0/1880805606 shutdown_connections 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.335+0000 7f7d05212740 1 -- 172.21.15.38:0/1880805606 wait complete. 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.335+0000 7f7d05212740 1 -- start start 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7d05212740 1 Processor -- start 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7d05212740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a93fd0 0x5579c3a9a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7d05212740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a94d40 0x5579c3a9aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7d05212740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a9d550 0x5579c3a9b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7d05212740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579c3aa0220 con 0x5579c3a9d550 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7d05212740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579c39b4d90 con 0x5579c3a93fd0 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7d05212740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5579c3aa03a0 con 0x5579c3a94d40 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7d05212740 10 client.0 _ll_register_callbacks cb 0x5579c370ca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:38:01.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7cfb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a9d550 0x5579c3a9b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7cfb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a9d550 0x5579c3a9b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53368/0 (socket says 172.21.15.38:53368) 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7cfb7fe640 1 -- 172.21.15.38:0/803637315 learned_addr learned my addr 172.21.15.38:0/803637315 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a93fd0 0x5579c3a9a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a94d40 0x5579c3a9aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7cfb7fe640 1 -- 172.21.15.38:0/803637315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a94d40 msgr2=0x5579c3a9aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a94d40 0x5579c3a9aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7cfb7fe640 1 -- 172.21.15.38:0/803637315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a93fd0 msgr2=0x5579c3a9a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a93fd0 0x5579c3a9a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7cfb7fe640 1 -- 172.21.15.38:0/803637315 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5579c37d9620 con 0x5579c3a9d550 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.336+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a94d40 0x5579c3a9aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a9d550 0x5579c3a9b030 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4012e00 tx=0x7f7ce40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89871321dfd84671 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7cdb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:38:01.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/803637315 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ce400fe00 con 0x5579c3a9d550 2024-03-26T06:38:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/803637315 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f7ce4004020 con 0x5579c3a9d550 2024-03-26T06:38:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7d05212740 1 -- 172.21.15.38:0/803637315 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cec02c880 con 0x5579c3a9d550 2024-03-26T06:38:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/803637315 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ce4027050 con 0x5579c3a9d550 2024-03-26T06:38:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7d05212740 10 client.15420 fetch_fsmap learned FSMap version 23 2024-03-26T06:38:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7d05212740 1 -- 172.21.15.38:0/803637315 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5579c3aac610 con 0x5579c3a9d550 2024-03-26T06:38:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/803637315 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cec02c880 con 0x5579c3a9d550 2024-03-26T06:38:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/803637315 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7ce4003d00 con 0x5579c3a9d550 2024-03-26T06:38:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7d05212740 10 client.15420 fetch_fsmap finished waiting for FSMap version 23 2024-03-26T06:38:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7d05212740 10 client.15420 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:38:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7d05212740 20 client.15420 populate_metadata read hostname 'smithi038' 2024-03-26T06:38:01.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7d05212740 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5579c3aada80 0x5579c3aafe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7d05212740 4 client.15420 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.337+0000 7f7d05212740 1 -- 172.21.15.38:0/803637315 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5579c3a05b30 con 0x5579c3aada80 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.338+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5579c3aada80 0x5579c3aafe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.338+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5579c3aada80 0x5579c3aafe70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e19c3e93ab334ab1 server_cookie=f1139302b3e68fb in_seq=0 out_seq=0 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.338+0000 7f7cdb7fe640 10 client.15420 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/803637315 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5579c3a05b30 con 0x5579c3aada80 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7cdb7fe640 10 client.15420 handle_command_reply: tid=0 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 1 client.15420 shutdown 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 1 -- 172.21.15.38:0/803637315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a9d550 msgr2=0x5579c3a9b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a9d550 0x5579c3a9b030 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4012e00 tx=0x7f7ce40046c0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 1 -- 172.21.15.38:0/803637315 shutdown_connections 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5579c3aada80 0x5579c3aafe70 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a9d550 0x5579c3a9b030 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a94d40 0x5579c3a9aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7cdb7fe640 0 client.15420 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 1 --2- 172.21.15.38:0/803637315 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a93fd0 0x5579c3a9a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 1 -- 172.21.15.38:0/803637315 >> 172.21.15.38:0/803637315 conn(0x5579c3a5b5b0 msgr2=0x5579c3a90a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 1 -- 172.21.15.38:0/803637315 shutdown_connections 2024-03-26T06:38:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 1 -- 172.21.15.38:0/803637315 wait complete. 2024-03-26T06:38:01.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.339+0000 7f7d05212740 20 client.15420 trim_cache size 0 max 16384 2024-03-26T06:38:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7d05212740 1 Processor -- start 2024-03-26T06:38:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7d05212740 1 -- start start 2024-03-26T06:38:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7d05212740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a93fd0 0x5579c3a9cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7d05212740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3aa4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7d05212740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 0x5579c3aa4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7d05212740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1af200 con 0x5579c3a94d40 2024-03-26T06:38:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7d05212740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1acda0 con 0x5579c3a93fd0 2024-03-26T06:38:01.361 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7d05212740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1aae40 con 0x5579c3a9d550 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7cfbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3aa4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7cfbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3aa4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53376/0 (socket says 172.21.15.38:53376) 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7cfbfff640 1 -- 172.21.15.38:0/1755172242 learned_addr learned my addr 172.21.15.38:0/1755172242 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/1755172242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 0x5579c3aa4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7cfbfff640 1 -- 172.21.15.38:0/1755172242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 msgr2=0x5579c3aa4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/1755172242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 0x5579c3aa4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7cfbfff640 1 -- 172.21.15.38:0/1755172242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a93fd0 msgr2=0x5579c3a9cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/1755172242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a93fd0 0x5579c3a9cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7cfbfff640 1 -- 172.21.15.38:0/1755172242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5579c3a932a0 con 0x5579c3a94d40 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.353+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/1755172242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 0x5579c3aa4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/1755172242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3aa4640 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7cec02c240 tx=0x7f7cec000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=745b7427eee1cb29 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:01.362 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7cf1ffb640 1 -- 172.21.15.38:0/1755172242 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec045040 con 0x5579c3a94d40 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7cf1ffb640 1 -- 172.21.15.38:0/1755172242 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cec04e020 con 0x5579c3a94d40 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7cf1ffb640 1 -- 172.21.15.38:0/1755172242 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec004ec0 con 0x5579c3a94d40 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7d05212740 1 -- 172.21.15.38:0/1755172242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 msgr2=0x5579c3aa4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7d05212740 1 --2- 172.21.15.38:0/1755172242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3aa4640 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f7cec02c240 tx=0x7f7cec000bc0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7d05212740 1 -- 172.21.15.38:0/1755172242 shutdown_connections 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7d05212740 1 --2- 172.21.15.38:0/1755172242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a9d550 0x5579c3aa4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7d05212740 1 --2- 172.21.15.38:0/1755172242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3aa4640 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7d05212740 1 --2- 172.21.15.38:0/1755172242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a93fd0 0x5579c3a9cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7d05212740 1 -- 172.21.15.38:0/1755172242 >> 172.21.15.38:0/1755172242 conn(0x5579c3a5b5b0 msgr2=0x5579c3a8e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7d05212740 1 -- 172.21.15.38:0/1755172242 shutdown_connections 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.354+0000 7f7d05212740 1 -- 172.21.15.38:0/1755172242 wait complete. 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7d05212740 1 -- start start 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7d05212740 1 Processor -- start 2024-03-26T06:38:01.363 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7d05212740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a93fd0 0x5579c3a97180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7d05212740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3a976c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7d05212740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a9d550 0x5579c3a97c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7d05212740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1ab440 con 0x5579c3a94d40 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7d05212740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1ac740 con 0x5579c3a9d550 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7d05212740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc1aba80 con 0x5579c3a93fd0 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7d05212740 10 client.0 _ll_register_callbacks cb 0x7f7cec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7cfbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3a976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7cfbfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3a976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:53384/0 (socket says 172.21.15.38:53384) 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.355+0000 7f7cfbfff640 1 -- 172.21.15.38:0/337955663 learned_addr learned my addr 172.21.15.38:0/337955663 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a93fd0 0x5579c3a97180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cfbfff640 1 -- 172.21.15.38:0/337955663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a93fd0 msgr2=0x5579c3a97180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a93fd0 0x5579c3a97180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.364 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cfbfff640 1 -- 172.21.15.38:0/337955663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a9d550 msgr2=0x5579c3a97c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a9d550 0x5579c3a97c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a9d550 0x5579c3a97c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cfbfff640 1 -- 172.21.15.38:0/337955663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5579c3b03f70 con 0x5579c3a94d40 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a9d550 0x5579c3a97c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cfbfff640 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3a976c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7cec02c820 tx=0x7f7cec02c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9d4e76409f6d0e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cdb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/337955663 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec045040 con 0x5579c3a94d40 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/337955663 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f7cec004ec0 con 0x5579c3a94d40 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7d05212740 1 -- 172.21.15.38:0/337955663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cec04e020 con 0x5579c3a94d40 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/337955663 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cec004980 con 0x5579c3a94d40 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7d05212740 10 client.15426 fetch_fsmap learned FSMap version 23 2024-03-26T06:38:01.365 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7d05212740 1 -- 172.21.15.38:0/337955663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5579c3b05130 con 0x5579c3a94d40 2024-03-26T06:38:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.356+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/337955663 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cec04e020 con 0x5579c3a94d40 2024-03-26T06:38:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.357+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/337955663 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7cec04aac0 con 0x5579c3a94d40 2024-03-26T06:38:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.357+0000 7f7d05212740 10 client.15426 fetch_fsmap finished waiting for FSMap version 23 2024-03-26T06:38:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.357+0000 7f7d05212740 10 client.15426 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:38:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.357+0000 7f7d05212740 20 client.15426 populate_metadata read hostname 'smithi038' 2024-03-26T06:38:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.357+0000 7f7d05212740 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5579c3b05350 0x5579c3b07740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.357+0000 7f7d05212740 4 client.15426 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:38:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.357+0000 7f7d05212740 1 -- 172.21.15.38:0/337955663 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5579c39ffe80 con 0x5579c3b05350 2024-03-26T06:38:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.357+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5579c3b05350 0x5579c3b07740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.358+0000 7f7cfb7fe640 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5579c3b05350 0x5579c3b07740 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ceba34910abcc974 server_cookie=f51419f2211a237a in_seq=0 out_seq=0 2024-03-26T06:38:01.366 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.358+0000 7f7cdb7fe640 10 client.15426 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:01.885 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.882+0000 7f7cdb7fe640 1 -- 172.21.15.38:0/337955663 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5579c39ffe80 con 0x5579c3b05350 2024-03-26T06:38:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.882+0000 7f7cdb7fe640 10 client.15426 handle_command_reply: tid=0 2024-03-26T06:38:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.882+0000 7f7d05212740 1 client.15426 shutdown 2024-03-26T06:38:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.882+0000 7f7d05212740 1 -- 172.21.15.38:0/337955663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 msgr2=0x5579c3a976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.882+0000 7f7d05212740 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3a976c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7cec02c820 tx=0x7f7cec02c880 comp rx=0 tx=0).stop 2024-03-26T06:38:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.883+0000 7f7d05212740 1 -- 172.21.15.38:0/337955663 shutdown_connections 2024-03-26T06:38:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.883+0000 7f7d05212740 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5579c3b05350 0x5579c3b07740 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.883+0000 7f7d05212740 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5579c3a9d550 0x5579c3a97c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.883+0000 7f7d05212740 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5579c3a94d40 0x5579c3a976c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.883+0000 7f7d05212740 1 --2- 172.21.15.38:0/337955663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5579c3a93fd0 0x5579c3a97180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.883+0000 7f7cdb7fe640 0 client.15426 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.883+0000 7f7d05212740 1 -- 172.21.15.38:0/337955663 >> 172.21.15.38:0/337955663 conn(0x5579c3a5b5b0 msgr2=0x5579c3a98db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.883+0000 7f7d05212740 1 -- 172.21.15.38:0/337955663 shutdown_connections 2024-03-26T06:38:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.883+0000 7f7d05212740 1 -- 172.21.15.38:0/337955663 wait complete. 2024-03-26T06:38:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.883+0000 7f7d05212740 20 client.15426 trim_cache size 0 max 16384 2024-03-26T06:38:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:38:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:38:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:38:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:38:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 -- 172.21.15.38:0/3637523555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f7ce80a2b30 msgr2=0x7f7ce80a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f7ce80a2b30 0x7f7ce80a4ff0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40ccd10 tx=0x7f7cf4090f80 comp rx=0 tx=0).stop 2024-03-26T06:38:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 -- 172.21.15.38:0/3637523555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cfc105950 msgr2=0x7f7cfc1978b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cfc105950 0x7f7cfc1978b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f7cf400eb10 tx=0x7f7cf400c2d0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 -- 172.21.15.38:0/3637523555 shutdown_connections 2024-03-26T06:38:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f7ce80a2b30 0x7f7ce80a4ff0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.888 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7cfc105950 0x7f7cfc1978b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7cfc100300 0x7f7cfc19dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 --2- 172.21.15.38:0/3637523555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7cfc0ff940 0x7f7cfc19d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:01.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 -- 172.21.15.38:0/3637523555 >> 172.21.15.38:0/3637523555 conn(0x7f7cfc0767b0 msgr2=0x7f7cfc1024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:01.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 -- 172.21.15.38:0/3637523555 shutdown_connections 2024-03-26T06:38:01.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:01.884+0000 7f7d02553640 1 -- 172.21.15.38:0/3637523555 wait complete. 2024-03-26T06:38:02.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:02 smithi038 ceph-mon[32265]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-26T06:38:02.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:02 smithi038 ceph-mon[32265]: mgrmap e25: x(active, since 28s), standbys: z, y 2024-03-26T06:38:02.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:02 smithi121 ceph-mon[36349]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-26T06:38:02.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:02 smithi121 ceph-mon[36349]: mgrmap e25: x(active, since 28s), standbys: z, y 2024-03-26T06:38:02.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-mon[36912]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-26T06:38:02.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-mon[36912]: mgrmap e25: x(active, since 28s), standbys: z, y 2024-03-26T06:38:03.029 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: *** Caught signal (Segmentation fault) ** 2024-03-26T06:38:03.029 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: in thread 7f59103e3700 thread_name:md_log_replay 2024-03-26T06:38:03.029 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: ceph version 19.0.0-1964-g7e98f5f6 (7e98f5f6b535defc7fa715b456904ffc0500aa9f) squid (dev) 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59217a4d20] 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 2: /usr/bin/ceph-mds(+0x503f02) [0x55725317cf02] 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x557253180793] 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 4: (MDLog::_replay_thread()+0x617) [0x5572531dc5a7] 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 5: (MDLog::ReplayThread::entry()+0x11) [0x5572531e50b1] 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f592179a1ca] 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 7: clone() 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 2024-03-26T06:38:02.722+0000 7f59103e3700 -1 *** Caught signal (Segmentation fault) ** 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: in thread 7f59103e3700 thread_name:md_log_replay 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: ceph version 19.0.0-1964-g7e98f5f6 (7e98f5f6b535defc7fa715b456904ffc0500aa9f) squid (dev) 2024-03-26T06:38:03.030 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59217a4d20] 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 2: /usr/bin/ceph-mds(+0x503f02) [0x55725317cf02] 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x557253180793] 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 4: (MDLog::_replay_thread()+0x617) [0x5572531dc5a7] 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 5: (MDLog::ReplayThread::entry()+0x11) [0x5572531e50b1] 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f592179a1ca] 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 7: clone() 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 0> 2024-03-26T06:38:02.722+0000 7f59103e3700 -1 *** Caught signal (Segmentation fault) ** 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: in thread 7f59103e3700 thread_name:md_log_replay 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: ceph version 19.0.0-1964-g7e98f5f6 (7e98f5f6b535defc7fa715b456904ffc0500aa9f) squid (dev) 2024-03-26T06:38:03.031 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59217a4d20] 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 2: /usr/bin/ceph-mds(+0x503f02) [0x55725317cf02] 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x557253180793] 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 4: (MDLog::_replay_thread()+0x617) [0x5572531dc5a7] 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 5: (MDLog::ReplayThread::entry()+0x11) [0x5572531e50b1] 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f592179a1ca] 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 7: clone() 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 0> 2024-03-26T06:38:02.722+0000 7f59103e3700 -1 *** Caught signal (Segmentation fault) ** 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: in thread 7f59103e3700 thread_name:md_log_replay 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: ceph version 19.0.0-1964-g7e98f5f6 (7e98f5f6b535defc7fa715b456904ffc0500aa9f) squid (dev) 2024-03-26T06:38:03.032 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f59217a4d20] 2024-03-26T06:38:03.033 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 2: /usr/bin/ceph-mds(+0x503f02) [0x55725317cf02] 2024-03-26T06:38:03.033 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 3: (MDSTableClient::got_journaled_ack(unsigned long)+0x173) [0x557253180793] 2024-03-26T06:38:03.033 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 4: (MDLog::_replay_thread()+0x617) [0x5572531dc5a7] 2024-03-26T06:38:03.033 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 5: (MDLog::ReplayThread::entry()+0x11) [0x5572531e50b1] 2024-03-26T06:38:03.033 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 6: /lib64/libpthread.so.0(+0x81ca) [0x7f592179a1ca] 2024-03-26T06:38:03.033 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 7: clone() 2024-03-26T06:38:03.033 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-26T06:38:03.033 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[68034]: 2024-03-26T06:38:03.033 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:02 smithi170 podman[71332]: 2024-03-26 06:38:02.78275559 +0000 UTC m=+0.021888308 container died 33d915e9dd0a2a1202eafbf23afdf4fdfee388288f5c3005d6c1179c301d8592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-26T06:38:03.918 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:03 smithi170 podman[71332]: 2024-03-26 06:38:03.418986736 +0000 UTC m=+0.658119436 container remove 33d915e9dd0a2a1202eafbf23afdf4fdfee388288f5c3005d6c1179c301d8592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:38:03.918 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:03 smithi170 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.c.service: Main process exited, code=exited, status=139/n/a 2024-03-26T06:38:03.918 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:03 smithi170 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.c.service: Failed with result 'exit-code'. 2024-03-26T06:38:04.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:04 smithi038 ceph-mon[32265]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-26T06:38:04.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:04 smithi121 ceph-mon[36349]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-26T06:38:04.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:04 smithi170 ceph-mon[36912]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-26T06:38:06.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:06 smithi121 ceph-mon[36349]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-26T06:38:06.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:06 smithi170 ceph-mon[36912]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-26T06:38:06.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:06 smithi038 ceph-mon[32265]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 24 KiB/s wr, 25 op/s 2024-03-26T06:38:08.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:08 smithi121 ceph-mon[36349]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-26T06:38:08.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:08 smithi170 ceph-mon[36912]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-26T06:38:08.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:08 smithi038 ceph-mon[32265]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-26T06:38:10.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:10 smithi121 ceph-mon[36349]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-26T06:38:10.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:10 smithi170 ceph-mon[36912]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-26T06:38:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:10 smithi038 ceph-mon[32265]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 24 KiB/s wr, 22 op/s 2024-03-26T06:38:12.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:12 smithi121 ceph-mon[36349]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-03-26T06:38:12.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:12 smithi170 ceph-mon[36912]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-03-26T06:38:12.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:12 smithi038 ceph-mon[32265]: pgmap v22: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 24 op/s 2024-03-26T06:38:13.995 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:13 smithi170 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.c.service: Scheduled restart job, restart counter is at 1. 2024-03-26T06:38:13.995 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:13 smithi170 systemd[1]: Stopped Ceph mds.c for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:38:13.995 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:13 smithi170 systemd[1]: Starting Ceph mds.c for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:38:13.995 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:13 smithi170 podman[71425]: 2024-03-26 06:38:13.89479783 +0000 UTC m=+0.112034498 container create b6177adb81ff81ef4df5d62305cc011235a4e83da6be22ad28c286ee839edb55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-26T06:38:13.995 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:13 smithi170 podman[71425]: 2024-03-26 06:38:13.801035542 +0000 UTC m=+0.018272225 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:38:14.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:14 smithi121 ceph-mon[36349]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-03-26T06:38:14.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:14 smithi170 ceph-mon[36912]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-03-26T06:38:14.418 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:14 smithi170 podman[71425]: 2024-03-26 06:38:14.105320617 +0000 UTC m=+0.322557288 container init b6177adb81ff81ef4df5d62305cc011235a4e83da6be22ad28c286ee839edb55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T06:38:14.418 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:14 smithi170 podman[71425]: 2024-03-26 06:38:14.108270319 +0000 UTC m=+0.325506991 container start b6177adb81ff81ef4df5d62305cc011235a4e83da6be22ad28c286ee839edb55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T06:38:14.418 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:14 smithi170 ceph-mds[71443]: -- 172.21.15.170:0/3799369550 <== mon.1 v2:172.21.15.170:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b0c1cb1a40 con 0x55b0c1c84c00 2024-03-26T06:38:14.418 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:14 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[71439]: starting mds.c at 2024-03-26T06:38:14.418 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:14 smithi170 bash[71425]: b6177adb81ff81ef4df5d62305cc011235a4e83da6be22ad28c286ee839edb55 2024-03-26T06:38:14.418 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:38:14 smithi170 systemd[1]: Started Ceph mds.c for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:38:14.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:14 smithi038 ceph-mon[32265]: pgmap v23: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 21 op/s 2024-03-26T06:38:14.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.985+0000 7f490ade1640 1 -- 172.21.15.38:0/2184944532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4904068970 msgr2=0x7f4904068df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:14.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.985+0000 7f490ade1640 1 --2- 172.21.15.38:0/2184944532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4904068970 0x7f4904068df0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009900 tx=0x7f48f002bcd0 comp rx=0 tx=0).stop 2024-03-26T06:38:14.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.985+0000 7f490ade1640 1 -- 172.21.15.38:0/2184944532 shutdown_connections 2024-03-26T06:38:14.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.985+0000 7f490ade1640 1 --2- 172.21.15.38:0/2184944532 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4904069330 0x7f490410e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:14.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.985+0000 7f490ade1640 1 --2- 172.21.15.38:0/2184944532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4904068970 0x7f4904068df0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:14.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.985+0000 7f490ade1640 1 --2- 172.21.15.38:0/2184944532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4904107400 0x7f49041077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:14.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.985+0000 7f490ade1640 1 -- 172.21.15.38:0/2184944532 >> 172.21.15.38:0/2184944532 conn(0x7f4904077040 msgr2=0x7f4904077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:14.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.985+0000 7f490ade1640 1 -- 172.21.15.38:0/2184944532 shutdown_connections 2024-03-26T06:38:14.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.985+0000 7f490ade1640 1 -- 172.21.15.38:0/2184944532 wait complete. 2024-03-26T06:38:14.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490ade1640 1 Processor -- start 2024-03-26T06:38:14.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490ade1640 1 -- start start 2024-03-26T06:38:14.990 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490ade1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4904068970 0x7f490419f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490ade1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4904069330 0x7f490419fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490ade1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4904107400 0x7f4904199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490ade1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4904111640 con 0x7f4904107400 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490ade1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49041114c0 con 0x7f4904069330 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490ade1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49041117c0 con 0x7f4904068970 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490937e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4904107400 0x7f4904199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490937e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4904107400 0x7f4904199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36970/0 (socket says 172.21.15.38:36970) 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490937e640 1 -- 172.21.15.38:0/2903452356 learned_addr learned my addr 172.21.15.38:0/2903452356 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f4908b7d640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4904068970 0x7f490419f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f4903fff640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4904069330 0x7f490419fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490937e640 1 -- 172.21.15.38:0/2903452356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4904068970 msgr2=0x7f490419f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490937e640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4904068970 0x7f490419f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:14.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490937e640 1 -- 172.21.15.38:0/2903452356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4904069330 msgr2=0x7f490419fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490937e640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4904069330 0x7f490419fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.986+0000 7f490937e640 1 -- 172.21.15.38:0/2903452356 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490419a240 con 0x7f4904107400 2024-03-26T06:38:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.987+0000 7f490937e640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4904107400 0x7f4904199ac0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00da40 tx=0x7f48fc00df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.987+0000 7f4901ffb640 1 -- 172.21.15.38:0/2903452356 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc017070 con 0x7f4904107400 2024-03-26T06:38:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.987+0000 7f490ade1640 1 -- 172.21.15.38:0/2903452356 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490419a530 con 0x7f4904107400 2024-03-26T06:38:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.987+0000 7f4901ffb640 1 -- 172.21.15.38:0/2903452356 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48fc003e50 con 0x7f4904107400 2024-03-26T06:38:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.987+0000 7f4901ffb640 1 -- 172.21.15.38:0/2903452356 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc010460 con 0x7f4904107400 2024-03-26T06:38:14.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.987+0000 7f490ade1640 1 -- 172.21.15.38:0/2903452356 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49041a67d0 con 0x7f4904107400 2024-03-26T06:38:14.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.989+0000 7f4901ffb640 1 -- 172.21.15.38:0/2903452356 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f48fc003970 con 0x7f4904107400 2024-03-26T06:38:14.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f490d9b6740 1 Processor -- start 2024-03-26T06:38:14.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f490d9b6740 1 -- start start 2024-03-26T06:38:14.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103418a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:14.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103418fd0 0x55a103419450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:14.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103422550 0x55a103424940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:14.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f490d9b6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a10337d5f0 con 0x55a103419d40 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f490d9b6740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a103326930 con 0x55a103418fd0 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f490d9b6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a103379690 con 0x55a103422550 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4903fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103418a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4903fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103418a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36978/0 (socket says 172.21.15.38:36978) 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4903fff640 1 -- 172.21.15.38:0/527988346 learned_addr learned my addr 172.21.15.38:0/527988346 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4901ffb640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f48f40a2b30 0x7f48f40a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4908b7d640 1 --2- 172.21.15.38:0/527988346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103418fd0 0x55a103419450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f490937e640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f48f40a2b30 0x7f48f40a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4903fff640 1 --2- 172.21.15.38:0/527988346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103422550 0x55a103424940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4901ffb640 1 -- 172.21.15.38:0/2903452356 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f48fc0ced70 con 0x7f4904107400 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4903fff640 1 -- 172.21.15.38:0/527988346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103422550 msgr2=0x55a103424940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4903fff640 1 --2- 172.21.15.38:0/527988346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103422550 0x55a103424940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4903fff640 1 -- 172.21.15.38:0/527988346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103418fd0 msgr2=0x55a103419450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4903fff640 1 --2- 172.21.15.38:0/527988346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103418fd0 0x55a103419450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4903fff640 1 -- 172.21.15.38:0/527988346 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a103092210 con 0x55a103419d40 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.990+0000 7f4908b7d640 1 --2- 172.21.15.38:0/527988346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103418fd0 0x55a103419450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.991+0000 7f4903fff640 1 --2- 172.21.15.38:0/527988346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103418a90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009900 tx=0x7f48f0005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599e37fe5e64cd97 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.991+0000 7f490937e640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f48f40a2b30 0x7f48f40a4ff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f48fc0cc390 tx=0x7f48fc06ce50 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.991+0000 7f48eeffd640 1 -- 172.21.15.38:0/527988346 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0049070 con 0x55a103419d40 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.991+0000 7f48eeffd640 1 -- 172.21.15.38:0/527988346 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48f002ce30 con 0x55a103419d40 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.991+0000 7f48eeffd640 1 -- 172.21.15.38:0/527988346 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0004ec0 con 0x55a103419d40 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 -- 172.21.15.38:0/527988346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 msgr2=0x55a103418a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 --2- 172.21.15.38:0/527988346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103418a90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009900 tx=0x7f48f0005e40 comp rx=0 tx=0).stop 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 -- 172.21.15.38:0/527988346 shutdown_connections 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 --2- 172.21.15.38:0/527988346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103422550 0x55a103424940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 --2- 172.21.15.38:0/527988346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103418fd0 0x55a103419450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 --2- 172.21.15.38:0/527988346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103418a90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 -- 172.21.15.38:0/527988346 >> 172.21.15.38:0/527988346 conn(0x55a1033e05b0 msgr2=0x55a103413cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 -- 172.21.15.38:0/527988346 shutdown_connections 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 -- 172.21.15.38:0/527988346 wait complete. 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 -- start start 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 Processor -- start 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103418fd0 0x55a10341f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103419d40 0x55a10341fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103422550 0x55a1034200b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.992+0000 7f490d9b6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a103425220 con 0x55a103422550 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f490d9b6740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a103339d90 con 0x55a103418fd0 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f490d9b6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1034253a0 con 0x55a103419d40 2024-03-26T06:38:15.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f490d9b6740 10 client.0 _ll_register_callbacks cb 0x55a103091a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f4903fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103422550 0x55a1034200b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f4903fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103422550 0x55a1034200b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36994/0 (socket says 172.21.15.38:36994) 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f4903fff640 1 -- 172.21.15.38:0/19761322 learned_addr learned my addr 172.21.15.38:0/19761322 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f4908b7d640 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103419d40 0x55a10341fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f4903fff640 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103418fd0 0x55a10341f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f4903fff640 1 -- 172.21.15.38:0/19761322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103419d40 msgr2=0x55a10341fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f4903fff640 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103419d40 0x55a10341fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f4903fff640 1 -- 172.21.15.38:0/19761322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103418fd0 msgr2=0x55a10341f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f4903fff640 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103418fd0 0x55a10341f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f4903fff640 1 -- 172.21.15.38:0/19761322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a10315e620 con 0x55a103422550 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f4903fff640 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103422550 0x55a1034200b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f48f002c210 tx=0x7f48f0004840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ee96373bd85393 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f48e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f48e3fff640 1 -- 172.21.15.38:0/19761322 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0049070 con 0x55a103422550 2024-03-26T06:38:15.003 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f48e3fff640 1 -- 172.21.15.38:0/19761322 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f48f002c950 con 0x55a103422550 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f490d9b6740 1 -- 172.21.15.38:0/19761322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48f002ce30 con 0x55a103422550 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f48e3fff640 1 -- 172.21.15.38:0/19761322 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48f002cb50 con 0x55a103422550 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f490d9b6740 10 client.15444 fetch_fsmap learned FSMap version 23 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.993+0000 7f490d9b6740 1 -- 172.21.15.38:0/19761322 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a1034315c0 con 0x55a103422550 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.994+0000 7f48e3fff640 1 -- 172.21.15.38:0/19761322 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48f002ce30 con 0x55a103422550 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.994+0000 7f48e3fff640 1 -- 172.21.15.38:0/19761322 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f48f00039c0 con 0x55a103422550 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.994+0000 7f490d9b6740 10 client.15444 fetch_fsmap finished waiting for FSMap version 23 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.994+0000 7f490d9b6740 10 client.15444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.994+0000 7f490d9b6740 20 client.15444 populate_metadata read hostname 'smithi038' 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.994+0000 7f490d9b6740 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55a103432910 0x55a103434d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.994+0000 7f490d9b6740 4 client.15444 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.994+0000 7f490d9b6740 1 -- 172.21.15.38:0/19761322 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a10338ab30 con 0x55a103432910 2024-03-26T06:38:15.004 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.994+0000 7f4908b7d640 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55a103432910 0x55a103434d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f4908b7d640 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55a103432910 0x55a103434d00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13656e9302e7850a server_cookie=58978df32974045c in_seq=0 out_seq=0 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f48e3fff640 10 client.15444 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f48e3fff640 1 -- 172.21.15.38:0/19761322 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a10338ab30 con 0x55a103432910 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f48e3fff640 10 client.15444 handle_command_reply: tid=0 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f490d9b6740 1 client.15444 shutdown 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f490d9b6740 1 -- 172.21.15.38:0/19761322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103422550 msgr2=0x55a1034200b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f490d9b6740 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103422550 0x55a1034200b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f48f002c210 tx=0x7f48f0004840 comp rx=0 tx=0).stop 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f490d9b6740 1 -- 172.21.15.38:0/19761322 shutdown_connections 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f490d9b6740 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55a103432910 0x55a103434d00 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f490d9b6740 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103422550 0x55a1034200b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f48e3fff640 0 client.15444 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f490d9b6740 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103419d40 0x55a10341fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f490d9b6740 1 --2- 172.21.15.38:0/19761322 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103418fd0 0x55a10341f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.995+0000 7f490d9b6740 1 -- 172.21.15.38:0/19761322 >> 172.21.15.38:0/19761322 conn(0x55a1033e05b0 msgr2=0x55a103415a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.996+0000 7f490d9b6740 1 -- 172.21.15.38:0/19761322 shutdown_connections 2024-03-26T06:38:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.996+0000 7f490d9b6740 1 -- 172.21.15.38:0/19761322 wait complete. 2024-03-26T06:38:15.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:14.996+0000 7f490d9b6740 20 client.15444 trim_cache size 0 max 16384 2024-03-26T06:38:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f490d9b6740 1 Processor -- start 2024-03-26T06:38:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f490d9b6740 1 -- start start 2024-03-26T06:38:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103418fd0 0x55a103421d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103419d40 0x55a103415820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103422550 0x55a10341c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f490d9b6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49041b12e0 con 0x55a103418fd0 2024-03-26T06:38:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f490d9b6740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49041aee80 con 0x55a103422550 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f490d9b6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49041acf20 con 0x55a103419d40 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f4903fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103418fd0 0x55a103421d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f4903fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103418fd0 0x55a103421d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36998/0 (socket says 172.21.15.38:36998) 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f4903fff640 1 -- 172.21.15.38:0/593482663 learned_addr learned my addr 172.21.15.38:0/593482663 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f4903fff640 1 -- 172.21.15.38:0/593482663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103419d40 msgr2=0x55a103415820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f4908b7d640 1 --2- 172.21.15.38:0/593482663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103419d40 0x55a103415820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f4903fff640 1 --2- 172.21.15.38:0/593482663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103419d40 0x55a103415820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f4903fff640 1 -- 172.21.15.38:0/593482663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103422550 msgr2=0x55a10341c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f4903fff640 1 --2- 172.21.15.38:0/593482663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103422550 0x55a10341c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f4903fff640 1 -- 172.21.15.38:0/593482663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a103418e00 con 0x55a103418fd0 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.023+0000 7f4903fff640 1 --2- 172.21.15.38:0/593482663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103418fd0 0x55a103421d60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f48f0003d90 tx=0x7f48f0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=248a30d546f0cc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.024+0000 7f48ee7fc640 1 -- 172.21.15.38:0/593482663 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f005d030 con 0x55a103418fd0 2024-03-26T06:38:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.024+0000 7f48ee7fc640 1 -- 172.21.15.38:0/593482663 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48f0046070 con 0x55a103418fd0 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.024+0000 7f48ee7fc640 1 -- 172.21.15.38:0/593482663 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f0049040 con 0x55a103418fd0 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.024+0000 7f490d9b6740 1 -- 172.21.15.38:0/593482663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103418fd0 msgr2=0x55a103421d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.024+0000 7f490d9b6740 1 --2- 172.21.15.38:0/593482663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103418fd0 0x55a103421d60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f48f0003d90 tx=0x7f48f0000bc0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.024+0000 7f490d9b6740 1 -- 172.21.15.38:0/593482663 shutdown_connections 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.024+0000 7f490d9b6740 1 --2- 172.21.15.38:0/593482663 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103422550 0x55a10341c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.024+0000 7f490d9b6740 1 --2- 172.21.15.38:0/593482663 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103419d40 0x55a103415820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.024+0000 7f490d9b6740 1 --2- 172.21.15.38:0/593482663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103418fd0 0x55a103421d60 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.024+0000 7f490d9b6740 1 -- 172.21.15.38:0/593482663 >> 172.21.15.38:0/593482663 conn(0x55a1033e05b0 msgr2=0x55a103415110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f490d9b6740 1 -- 172.21.15.38:0/593482663 shutdown_connections 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f490d9b6740 1 -- 172.21.15.38:0/593482663 wait complete. 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f490d9b6740 1 -- start start 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f490d9b6740 1 Processor -- start 2024-03-26T06:38:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103418fd0 0x55a103435930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103435e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f490d9b6740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103422550 0x55a1034363b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f490d9b6740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49041ad520 con 0x55a103419d40 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f490d9b6740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49041ae820 con 0x55a103422550 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f490d9b6740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49041adb60 con 0x55a103418fd0 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f490d9b6740 10 client.0 _ll_register_callbacks cb 0x7f48f0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f4908b7d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103435e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f4908b7d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103435e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37000/0 (socket says 172.21.15.38:37000) 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.025+0000 7f4908b7d640 1 -- 172.21.15.38:0/32759258 learned_addr learned my addr 172.21.15.38:0/32759258 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f4903fff640 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103418fd0 0x55a103435930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f4903fff640 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103422550 0x55a1034363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f4908b7d640 1 -- 172.21.15.38:0/32759258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103418fd0 msgr2=0x55a103435930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f4908b7d640 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103418fd0 0x55a103435930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f4908b7d640 1 -- 172.21.15.38:0/32759258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103422550 msgr2=0x55a1034363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f4908b7d640 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103422550 0x55a1034363b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f4908b7d640 1 -- 172.21.15.38:0/32759258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a103488c90 con 0x55a103419d40 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f4903fff640 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103422550 0x55a1034363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f4908b7d640 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103435e70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55a10341ee90 tx=0x7f48f80096f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245bb7c5f18e19d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f48e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f48e3fff640 1 -- 172.21.15.38:0/32759258 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f801f400 con 0x55a103419d40 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f48e3fff640 1 -- 172.21.15.38:0/32759258 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f48f801f5a0 con 0x55a103419d40 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f490d9b6740 1 -- 172.21.15.38:0/32759258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48f0046070 con 0x55a103419d40 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f48e3fff640 1 -- 172.21.15.38:0/32759258 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48f802e050 con 0x55a103419d40 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f490d9b6740 10 client.15450 fetch_fsmap learned FSMap version 23 2024-03-26T06:38:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f490d9b6740 1 -- 172.21.15.38:0/32759258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55a103489ff0 con 0x55a103419d40 2024-03-26T06:38:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.026+0000 7f48e3fff640 1 -- 172.21.15.38:0/32759258 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48f0046070 con 0x55a103419d40 2024-03-26T06:38:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.027+0000 7f48e3fff640 1 -- 172.21.15.38:0/32759258 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f48f80273e0 con 0x55a103419d40 2024-03-26T06:38:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.027+0000 7f490d9b6740 10 client.15450 fetch_fsmap finished waiting for FSMap version 23 2024-03-26T06:38:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.027+0000 7f490d9b6740 10 client.15450 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:38:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.027+0000 7f490d9b6740 20 client.15450 populate_metadata read hostname 'smithi038' 2024-03-26T06:38:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.027+0000 7f490d9b6740 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55a10348a750 0x55a10348cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.027+0000 7f490d9b6740 4 client.15450 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:38:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.027+0000 7f490d9b6740 1 -- 172.21.15.38:0/32759258 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a103384e80 con 0x55a10348a750 2024-03-26T06:38:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.027+0000 7f4903fff640 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55a10348a750 0x55a10348cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.027+0000 7f4903fff640 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55a10348a750 0x55a10348cb70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d7f89a94a4fe768 server_cookie=394791df2e564729 in_seq=0 out_seq=0 2024-03-26T06:38:15.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.027+0000 7f48e3fff640 10 client.15450 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f48e3fff640 1 -- 172.21.15.38:0/32759258 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a103384e80 con 0x55a10348a750 2024-03-26T06:38:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f48e3fff640 10 client.15450 handle_command_reply: tid=0 2024-03-26T06:38:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 1 client.15450 shutdown 2024-03-26T06:38:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 1 -- 172.21.15.38:0/32759258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 msgr2=0x55a103435e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103435e70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55a10341ee90 tx=0x7f48f80096f0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 1 -- 172.21.15.38:0/32759258 shutdown_connections 2024-03-26T06:38:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55a10348a750 0x55a10348cb70 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55a103422550 0x55a1034363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55a103419d40 0x55a103435e70 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f48e3fff640 0 client.15450 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 1 --2- 172.21.15.38:0/32759258 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55a103418fd0 0x55a103435930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 1 -- 172.21.15.38:0/32759258 >> 172.21.15.38:0/32759258 conn(0x55a1033e05b0 msgr2=0x55a10341daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 1 -- 172.21.15.38:0/32759258 shutdown_connections 2024-03-26T06:38:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 1 -- 172.21.15.38:0/32759258 wait complete. 2024-03-26T06:38:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.035+0000 7f490d9b6740 20 client.15450 trim_cache size 0 max 16384 2024-03-26T06:38:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:38:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:38:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:38:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:38:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 -- 172.21.15.38:0/2903452356 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f48f40a2b30 msgr2=0x7f48f40a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f48f40a2b30 0x7f48f40a4ff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f48fc0cc390 tx=0x7f48fc06ce50 comp rx=0 tx=0).stop 2024-03-26T06:38:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 -- 172.21.15.38:0/2903452356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4904107400 msgr2=0x7f4904199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4904107400 0x7f4904199ac0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00da40 tx=0x7f48fc00df00 comp rx=0 tx=0).stop 2024-03-26T06:38:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 -- 172.21.15.38:0/2903452356 shutdown_connections 2024-03-26T06:38:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f48f40a2b30 0x7f48f40a4ff0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4904107400 0x7f4904199ac0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4904069330 0x7f490419fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 --2- 172.21.15.38:0/2903452356 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4904068970 0x7f490419f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 -- 172.21.15.38:0/2903452356 >> 172.21.15.38:0/2903452356 conn(0x7f4904077040 msgr2=0x7f490410c530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 -- 172.21.15.38:0/2903452356 shutdown_connections 2024-03-26T06:38:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:15.036+0000 7f490ade1640 1 -- 172.21.15.38:0/2903452356 wait complete. 2024-03-26T06:38:15.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:15 smithi121 ceph-mon[36349]: daemon mds.c restarted 2024-03-26T06:38:15.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:15 smithi170 ceph-mon[36912]: daemon mds.c restarted 2024-03-26T06:38:15.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:15 smithi038 ceph-mon[32265]: daemon mds.c restarted 2024-03-26T06:38:16.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:16 smithi121 ceph-mon[36349]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-03-26T06:38:16.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:16 smithi121 ceph-mon[36349]: osdmap e90: 12 total, 12 up, 12 in 2024-03-26T06:38:16.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:16 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.170:6836/1404647281,v1:172.21.15.170:6837/1404647281] up:boot 2024-03-26T06:38:16.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:16 smithi121 ceph-mon[36349]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-26T06:38:16.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:16 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:38:16.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:16 smithi121 ceph-mon[36349]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:38:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:16 smithi170 ceph-mon[36912]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-03-26T06:38:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:16 smithi170 ceph-mon[36912]: osdmap e90: 12 total, 12 up, 12 in 2024-03-26T06:38:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:16 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.170:6836/1404647281,v1:172.21.15.170:6837/1404647281] up:boot 2024-03-26T06:38:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:16 smithi170 ceph-mon[36912]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-26T06:38:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:16 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:38:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:16 smithi170 ceph-mon[36912]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:38:16.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:16 smithi038 ceph-mon[32265]: pgmap v24: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-03-26T06:38:16.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:16 smithi038 ceph-mon[32265]: osdmap e90: 12 total, 12 up, 12 in 2024-03-26T06:38:16.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:16 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.170:6836/1404647281,v1:172.21.15.170:6837/1404647281] up:boot 2024-03-26T06:38:16.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:16 smithi038 ceph-mon[32265]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-03-26T06:38:16.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:16 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:38:16.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:16 smithi038 ceph-mon[32265]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:38:18.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:18 smithi121 ceph-mon[36349]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-26T06:38:18.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:18 smithi170 ceph-mon[36912]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-26T06:38:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:18 smithi038 ceph-mon[32265]: pgmap v26: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-26T06:38:20.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:20 smithi121 ceph-mon[36349]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-26T06:38:20.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:20 smithi170 ceph-mon[36912]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-26T06:38:20.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:20 smithi038 ceph-mon[32265]: pgmap v27: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.9 KiB/s wr, 5 op/s 2024-03-26T06:38:22.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:22 smithi121 ceph-mon[36349]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-26T06:38:22.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:22 smithi170 ceph-mon[36912]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-26T06:38:22.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:22 smithi038 ceph-mon[32265]: pgmap v28: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-26T06:38:24.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:24 smithi121 ceph-mon[36349]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-26T06:38:24.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:24 smithi170 ceph-mon[36912]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-26T06:38:24.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:24 smithi038 ceph-mon[32265]: pgmap v29: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-26T06:38:26.386 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:26 smithi038 ceph-mon[32265]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-26T06:38:26.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:26 smithi121 ceph-mon[36349]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-26T06:38:26.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:26 smithi170 ceph-mon[36912]: pgmap v30: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-26T06:38:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.130+0000 7f66e9612640 1 -- 172.21.15.38:0/304798883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4100440 msgr2=0x7f66e41008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.130+0000 7f66e9612640 1 --2- 172.21.15.38:0/304798883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4100440 0x7f66e41008c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f66d8009900 tx=0x7f66d802bc30 comp rx=0 tx=0).stop 2024-03-26T06:38:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.130+0000 7f66e9612640 1 -- 172.21.15.38:0/304798883 shutdown_connections 2024-03-26T06:38:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.130+0000 7f66e9612640 1 --2- 172.21.15.38:0/304798883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66e4100e00 0x7f66e410d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.130+0000 7f66e9612640 1 --2- 172.21.15.38:0/304798883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4100440 0x7f66e41008c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.133 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.130+0000 7f66e9612640 1 --2- 172.21.15.38:0/304798883 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e4105190 0x7f66e4105570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.130+0000 7f66e9612640 1 -- 172.21.15.38:0/304798883 >> 172.21.15.38:0/304798883 conn(0x7f66e4077040 msgr2=0x7f66e4077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:28.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.130+0000 7f66e9612640 1 -- 172.21.15.38:0/304798883 shutdown_connections 2024-03-26T06:38:28.134 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.130+0000 7f66e9612640 1 -- 172.21.15.38:0/304798883 wait complete. 2024-03-26T06:38:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.131+0000 7f66e9612640 1 Processor -- start 2024-03-26T06:38:28.136 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.131+0000 7f66e9612640 1 -- start start 2024-03-26T06:38:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.131+0000 7f66e9612640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4100440 0x7f66e4198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.131+0000 7f66e9612640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66e4100e00 0x7f66e4198f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.131+0000 7f66e9612640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e4105190 0x7f66e4199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.131+0000 7f66e9612640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e41106a0 con 0x7f66e4100440 2024-03-26T06:38:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.131+0000 7f66e9612640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e4110520 con 0x7f66e4100e00 2024-03-26T06:38:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.131+0000 7f66e9612640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e4110820 con 0x7f66e4105190 2024-03-26T06:38:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.131+0000 7f66e2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4100440 0x7f66e4198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4100440 0x7f66e4198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50502/0 (socket says 172.21.15.38:50502) 2024-03-26T06:38:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e2ffd640 1 -- 172.21.15.38:0/3565401184 learned_addr learned my addr 172.21.15.38:0/3565401184 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:28.137 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e27fc640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66e4100e00 0x7f66e4198f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e2ffd640 1 -- 172.21.15.38:0/3565401184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e4105190 msgr2=0x7f66e4199500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e4105190 0x7f66e4199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e2ffd640 1 -- 172.21.15.38:0/3565401184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66e4100e00 msgr2=0x7f66e4198f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66e4100e00 0x7f66e4198f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e2ffd640 1 -- 172.21.15.38:0/3565401184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e41a3e40 con 0x7f66e4100440 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e27fc640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66e4100e00 0x7f66e4198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4100440 0x7f66e4198a40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f66d400e9f0 tx=0x7f66d400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66cbfff640 1 -- 172.21.15.38:0/3565401184 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d400cd80 con 0x7f66e4100440 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e9612640 1 -- 172.21.15.38:0/3565401184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e41a4130 con 0x7f66e4100440 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66cbfff640 1 -- 172.21.15.38:0/3565401184 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66d4005500 con 0x7f66e4100440 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66cbfff640 1 -- 172.21.15.38:0/3565401184 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4010690 con 0x7f66e4100440 2024-03-26T06:38:28.138 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.132+0000 7f66e9612640 1 -- 172.21.15.38:0/3565401184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e41a4650 con 0x7f66e4100440 2024-03-26T06:38:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.135+0000 7f66ec1f5740 1 Processor -- start 2024-03-26T06:38:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.135+0000 7f66ec1f5740 1 -- start start 2024-03-26T06:38:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66cbfff640 1 -- 172.21.15.38:0/3565401184 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f66d4010830 con 0x7f66e4100440 2024-03-26T06:38:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf32a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf33450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf3e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebfae975f0 con 0x55ebfaf3c550 2024-03-26T06:38:28.144 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebfae40930 con 0x55ebfaf33d40 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebfae93690 con 0x55ebfaf32fd0 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66e27fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf3e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66e27fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf3e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50516/0 (socket says 172.21.15.38:50516) 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66e27fc640 1 -- 172.21.15.38:0/2969288830 learned_addr learned my addr 172.21.15.38:0/2969288830 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66e37fe640 1 --2- 172.21.15.38:0/2969288830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf32a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/2969288830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf33450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66e27fc640 1 -- 172.21.15.38:0/2969288830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf32fd0 msgr2=0x55ebfaf33450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66e27fc640 1 --2- 172.21.15.38:0/2969288830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf33450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66e27fc640 1 -- 172.21.15.38:0/2969288830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 msgr2=0x55ebfaf32a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66e27fc640 1 --2- 172.21.15.38:0/2969288830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf32a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66e27fc640 1 -- 172.21.15.38:0/2969288830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebfabac210 con 0x55ebfaf3c550 2024-03-26T06:38:28.145 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66e37fe640 1 --2- 172.21.15.38:0/2969288830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf32a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.136+0000 7f66cbfff640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f66cc0a2b30 0x7f66cc0a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.137+0000 7f66cbfff640 1 -- 172.21.15.38:0/3565401184 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f66d40cf620 con 0x7f66e4100440 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.137+0000 7f66e27fc640 1 --2- 172.21.15.38:0/2969288830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf3e940 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f66d802efd0 tx=0x7f66d802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd92cccd6b649c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.137+0000 7f66e37fe640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f66cc0a2b30 0x7f66cc0a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.137+0000 7f66c97fa640 1 -- 172.21.15.38:0/2969288830 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8042070 con 0x55ebfaf3c550 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.137+0000 7f66c97fa640 1 -- 172.21.15.38:0/2969288830 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66d8004240 con 0x55ebfaf3c550 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.137+0000 7f66c97fa640 1 -- 172.21.15.38:0/2969288830 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d80046c0 con 0x55ebfaf3c550 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 -- 172.21.15.38:0/2969288830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 msgr2=0x55ebfaf3e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66e37fe640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f66cc0a2b30 0x7f66cc0a4ff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0099a0 tx=0x7f66dc009200 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/2969288830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf3e940 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f66d802efd0 tx=0x7f66d802c5d0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 -- 172.21.15.38:0/2969288830 shutdown_connections 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/2969288830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf3e940 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.146 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/2969288830 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf33450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/2969288830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf32a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 -- 172.21.15.38:0/2969288830 >> 172.21.15.38:0/2969288830 conn(0x55ebfaefa5b0 msgr2=0x55ebfaf2dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 -- 172.21.15.38:0/2969288830 shutdown_connections 2024-03-26T06:38:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 -- 172.21.15.38:0/2969288830 wait complete. 2024-03-26T06:38:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 -- start start 2024-03-26T06:38:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 Processor -- start 2024-03-26T06:38:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf39530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf39a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.147 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.138+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf3c550 0x55ebfaf39fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebfaf3f220 con 0x55ebfaf32fd0 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebfae53d90 con 0x55ebfaf33d40 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ebfaf3f3a0 con 0x55ebfaf3c550 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66ec1f5740 10 client.0 _ll_register_callbacks cb 0x55ebfababa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66e2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf39530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66e2ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf39530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50532/0 (socket says 172.21.15.38:50532) 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66e2ffd640 1 -- 172.21.15.38:0/1654861004 learned_addr learned my addr 172.21.15.38:0/1654861004 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66e27fc640 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf39a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66e37fe640 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf3c550 0x55ebfaf39fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66e2ffd640 1 -- 172.21.15.38:0/1654861004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf3c550 msgr2=0x55ebfaf39fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf3c550 0x55ebfaf39fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66e2ffd640 1 -- 172.21.15.38:0/1654861004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 msgr2=0x55ebfaf39a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:28.148 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf39a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66e2ffd640 1 -- 172.21.15.38:0/1654861004 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ebfac78620 con 0x55ebfaf32fd0 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf39530 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f66d400c6a0 tx=0x7f66d40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1941cf497a18e34e server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.139+0000 7f66c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.140+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1654861004 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d409b560 con 0x55ebfaf32fd0 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.140+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1654861004 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f66d8004240 con 0x55ebfaf32fd0 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.140+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1654861004 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f66d40a3d30 con 0x55ebfaf32fd0 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.140+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1654861004 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66d409b380 con 0x55ebfaf32fd0 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.140+0000 7f66ec1f5740 10 client.15462 fetch_fsmap learned FSMap version 25 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.140+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1654861004 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ebfaf4b560 con 0x55ebfaf32fd0 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.140+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1654861004 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66d8004240 con 0x55ebfaf32fd0 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.141+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1654861004 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f66d40ccec0 con 0x55ebfaf32fd0 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.141+0000 7f66ec1f5740 10 client.15462 fetch_fsmap finished waiting for FSMap version 25 2024-03-26T06:38:28.149 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.141+0000 7f66ec1f5740 10 client.15462 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:38:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.141+0000 7f66ec1f5740 20 client.15462 populate_metadata read hostname 'smithi038' 2024-03-26T06:38:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.141+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ebfaf4c9d0 0x55ebfaf4edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.141+0000 7f66ec1f5740 4 client.15462 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:38:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.141+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1654861004 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ebfaea4b30 con 0x55ebfaf4c9d0 2024-03-26T06:38:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.141+0000 7f66e27fc640 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ebfaf4c9d0 0x55ebfaf4edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.141+0000 7f66c6ff5640 10 client.15462 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:28.150 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.141+0000 7f66e27fc640 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ebfaf4c9d0 0x55ebfaf4edc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a143c7c44cf4905 server_cookie=8670dc5c9d655daf in_seq=0 out_seq=0 2024-03-26T06:38:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1654861004 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ebfaea4b30 con 0x55ebfaf4c9d0 2024-03-26T06:38:28.151 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66c6ff5640 10 client.15462 handle_command_reply: tid=0 2024-03-26T06:38:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 1 client.15462 shutdown 2024-03-26T06:38:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1654861004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf32fd0 msgr2=0x55ebfaf39530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf39530 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f66d400c6a0 tx=0x7f66d40057b0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1654861004 shutdown_connections 2024-03-26T06:38:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ebfaf4c9d0 0x55ebfaf4edc0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.152 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf3c550 0x55ebfaf39fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf39a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66c6ff5640 0 client.15462 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1654861004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf39530 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1654861004 >> 172.21.15.38:0/1654861004 conn(0x55ebfaefa5b0 msgr2=0x55ebfaf2fa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1654861004 shutdown_connections 2024-03-26T06:38:28.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1654861004 wait complete. 2024-03-26T06:38:28.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.142+0000 7f66ec1f5740 20 client.15462 trim_cache size 0 max 16384 2024-03-26T06:38:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66ec1f5740 1 Processor -- start 2024-03-26T06:38:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66ec1f5740 1 -- start start 2024-03-26T06:38:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf3bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf33d40 0x55ebfaf367b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf3c550 0x55ebfaf36cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e41af2d0 con 0x55ebfaf33d40 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e41ace70 con 0x55ebfaf32fd0 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e41aaf10 con 0x55ebfaf3c550 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66e27fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf33d40 0x55ebfaf367b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66e27fc640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf33d40 0x55ebfaf367b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50546/0 (socket says 172.21.15.38:50546) 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66e27fc640 1 -- 172.21.15.38:0/1677907017 learned_addr learned my addr 172.21.15.38:0/1677907017 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/1677907017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf3bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66e27fc640 1 -- 172.21.15.38:0/1677907017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf3c550 msgr2=0x55ebfaf36cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66e27fc640 1 --2- 172.21.15.38:0/1677907017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf3c550 0x55ebfaf36cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66e27fc640 1 -- 172.21.15.38:0/1677907017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf32fd0 msgr2=0x55ebfaf3bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66e27fc640 1 --2- 172.21.15.38:0/1677907017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf3bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66e27fc640 1 -- 172.21.15.38:0/1677907017 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ebfaf29590 con 0x55ebfaf33d40 2024-03-26T06:38:28.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.162+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/1677907017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf3bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:38:28.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.163+0000 7f66e27fc640 1 --2- 172.21.15.38:0/1677907017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf33d40 0x55ebfaf367b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f66d80053f0 tx=0x7f66d8005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568ca8ca7f09c969 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:28.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.163+0000 7f66c8ff9640 1 -- 172.21.15.38:0/1677907017 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8042040 con 0x55ebfaf33d40 2024-03-26T06:38:28.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.163+0000 7f66c8ff9640 1 -- 172.21.15.38:0/1677907017 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66d802c6c0 con 0x55ebfaf33d40 2024-03-26T06:38:28.167 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.163+0000 7f66c8ff9640 1 -- 172.21.15.38:0/1677907017 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d8053740 con 0x55ebfaf33d40 2024-03-26T06:38:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.163+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1677907017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf33d40 msgr2=0x55ebfaf367b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:28.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.163+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1677907017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf33d40 0x55ebfaf367b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f66d80053f0 tx=0x7f66d8005420 comp rx=0 tx=0).stop 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.163+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1677907017 shutdown_connections 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.163+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1677907017 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf3c550 0x55ebfaf36cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.163+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1677907017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf33d40 0x55ebfaf367b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.163+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1677907017 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf3bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.163+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1677907017 >> 172.21.15.38:0/1677907017 conn(0x55ebfaefa5b0 msgr2=0x55ebfaf37e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1677907017 shutdown_connections 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1677907017 wait complete. 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66ec1f5740 1 -- start start 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66ec1f5740 1 Processor -- start 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf429e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf42f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66ec1f5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf9cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e41ab510 con 0x55ebfaf3c550 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e41ac810 con 0x55ebfaf33d40 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66ec1f5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e41abb50 con 0x55ebfaf32fd0 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66e37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf9cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66e27fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf42f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66e37fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf9cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50548/0 (socket says 172.21.15.38:50548) 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66e27fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf42f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:51558/0 (socket says 172.21.15.38:51558) 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66e37fe640 1 -- 172.21.15.38:0/1369104313 learned_addr learned my addr 172.21.15.38:0/1369104313 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.164+0000 7f66ec1f5740 10 client.0 _ll_register_callbacks cb 0x7f66dc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66e37fe640 1 -- 172.21.15.38:0/1369104313 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf32fd0 msgr2=0x55ebfaf429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66e37fe640 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf429e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66e37fe640 1 -- 172.21.15.38:0/1369104313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 msgr2=0x55ebfaf42f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:28.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66e37fe640 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf42f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66e37fe640 1 -- 172.21.15.38:0/1369104313 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ebfaf9d520 con 0x55ebfaf3c550 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66e37fe640 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf9cee0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f66dc003e90 tx=0x7f66dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901d00d2ffd0955c server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66c6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1369104313 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66dc004b20 con 0x55ebfaf3c550 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1369104313 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f66dc004cc0 con 0x55ebfaf3c550 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1369104313 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ebfae9ee80 con 0x55ebfaf3c550 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1369104313 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f66dc00f5c0 con 0x55ebfaf3c550 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66ec1f5740 10 client.15468 fetch_fsmap learned FSMap version 25 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.165+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1369104313 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55ebfafa4810 con 0x55ebfaf3c550 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.166+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1369104313 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66dc00fc40 con 0x55ebfaf3c550 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.166+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1369104313 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f66dc027a30 con 0x55ebfaf3c550 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.166+0000 7f66ec1f5740 10 client.15468 fetch_fsmap finished waiting for FSMap version 25 2024-03-26T06:38:28.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.166+0000 7f66ec1f5740 10 client.15468 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:38:28.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.166+0000 7f66ec1f5740 20 client.15468 populate_metadata read hostname 'smithi038' 2024-03-26T06:38:28.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.166+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ebfafa4fc0 0x55ebfafa73e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:28.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.166+0000 7f66ec1f5740 4 client.15468 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:38:28.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.166+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1369104313 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ebfae9ee80 con 0x55ebfafa4fc0 2024-03-26T06:38:28.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.166+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ebfafa4fc0 0x55ebfafa73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:28.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.167+0000 7f66c6ff5640 10 client.15468 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:28.174 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.167+0000 7f66e2ffd640 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ebfafa4fc0 0x55ebfafa73e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f32b567b77d6c30 server_cookie=f7f0dd87ed0071c6 in_seq=0 out_seq=0 2024-03-26T06:38:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66c6ff5640 1 -- 172.21.15.38:0/1369104313 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ebfae9ee80 con 0x55ebfafa4fc0 2024-03-26T06:38:28.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66c6ff5640 10 client.15468 handle_command_reply: tid=0 2024-03-26T06:38:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66ec1f5740 1 client.15468 shutdown 2024-03-26T06:38:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1369104313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 msgr2=0x55ebfaf9cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf9cee0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f66dc003e90 tx=0x7f66dc004a40 comp rx=0 tx=0).stop 2024-03-26T06:38:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1369104313 shutdown_connections 2024-03-26T06:38:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ebfafa4fc0 0x55ebfafa73e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ebfaf3c550 0x55ebfaf9cee0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ebfaf33d40 0x55ebfaf42f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66c6ff5640 0 client.15468 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:28.176 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66ec1f5740 1 --2- 172.21.15.38:0/1369104313 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ebfaf32fd0 0x55ebfaf429e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1369104313 >> 172.21.15.38:0/1369104313 conn(0x55ebfaefa5b0 msgr2=0x55ebfaf2c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:28.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.170+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1369104313 shutdown_connections 2024-03-26T06:38:28.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66ec1f5740 1 -- 172.21.15.38:0/1369104313 wait complete. 2024-03-26T06:38:28.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66ec1f5740 20 client.15468 trim_cache size 0 max 16384 2024-03-26T06:38:28.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:38:28.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:38:28.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:38:28.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:38:28.178 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 -- 172.21.15.38:0/3565401184 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f66cc0a2b30 msgr2=0x7f66cc0a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f66cc0a2b30 0x7f66cc0a4ff0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f66dc0099a0 tx=0x7f66dc009200 comp rx=0 tx=0).stop 2024-03-26T06:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 -- 172.21.15.38:0/3565401184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4100440 msgr2=0x7f66e4198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4100440 0x7f66e4198a40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f66d400e9f0 tx=0x7f66d400edf0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 -- 172.21.15.38:0/3565401184 shutdown_connections 2024-03-26T06:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f66cc0a2b30 0x7f66cc0a4ff0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f66e4105190 0x7f66e4199500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66e4100e00 0x7f66e4198f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 --2- 172.21.15.38:0/3565401184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f66e4100440 0x7f66e4198a40 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 -- 172.21.15.38:0/3565401184 >> 172.21.15.38:0/3565401184 conn(0x7f66e4077040 msgr2=0x7f66e40fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 -- 172.21.15.38:0/3565401184 shutdown_connections 2024-03-26T06:38:28.179 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:28.171+0000 7f66e9612640 1 -- 172.21.15.38:0/3565401184 wait complete. 2024-03-26T06:38:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:28 smithi121 ceph-mon[36349]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-26T06:38:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:28 smithi170 ceph-mon[36912]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-26T06:38:28.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:28 smithi038 ceph-mon[32265]: pgmap v31: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-26T06:38:30.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:30 smithi121 ceph-mon[36349]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-26T06:38:30.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:30 smithi170 ceph-mon[36912]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-26T06:38:30.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:30 smithi038 ceph-mon[32265]: pgmap v32: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-26T06:38:32.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:32 smithi121 ceph-mon[36349]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-26T06:38:32.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:32 smithi170 ceph-mon[36912]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-26T06:38:32.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:32 smithi038 ceph-mon[32265]: pgmap v33: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-26T06:38:34.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:34 smithi121 ceph-mon[36349]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:34.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:34 smithi170 ceph-mon[36912]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:34.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:34 smithi038 ceph-mon[32265]: pgmap v34: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:36 smithi121 ceph-mon[36349]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:36.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:36 smithi170 ceph-mon[36912]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:36.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:36 smithi038 ceph-mon[32265]: pgmap v35: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:38.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:38 smithi170 ceph-mon[36912]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:38.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:38:38.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:38:38.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:38:38.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:38 smithi038 ceph-mon[32265]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:38.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:38:38.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:38:38.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:38:38.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:38 smithi121 ceph-mon[36349]: pgmap v36: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:38.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:38:38.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:38:38.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:38:39.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:39 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:38:39.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:39 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:38:39.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:39 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:38:39.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:39 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:38:39.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:39 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:38:39.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:39 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:38:39.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:39 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:38:39.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:39 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:38:39.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:39 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:38:40.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:40 smithi170 ceph-mon[36912]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:40.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:40 smithi038 ceph-mon[32265]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:40.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:40 smithi121 ceph-mon[36349]: pgmap v37: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.308+0000 7f62a7829640 1 -- 172.21.15.38:0/3540455587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a0100e00 msgr2=0x7f62a010d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.308+0000 7f62a7829640 1 --2- 172.21.15.38:0/3540455587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a0100e00 0x7f62a010d020 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f6298009ca0 tx=0x7f629802c0f0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.308+0000 7f62a7829640 1 -- 172.21.15.38:0/3540455587 shutdown_connections 2024-03-26T06:38:41.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.308+0000 7f62a7829640 1 --2- 172.21.15.38:0/3540455587 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a0100e00 0x7f62a010d020 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.308+0000 7f62a7829640 1 --2- 172.21.15.38:0/3540455587 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a0100440 0x7f62a01008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.308+0000 7f62a7829640 1 --2- 172.21.15.38:0/3540455587 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a0105190 0x7f62a0105570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.308+0000 7f62a7829640 1 -- 172.21.15.38:0/3540455587 >> 172.21.15.38:0/3540455587 conn(0x7f62a0077040 msgr2=0x7f62a0077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.308+0000 7f62a7829640 1 -- 172.21.15.38:0/3540455587 shutdown_connections 2024-03-26T06:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.308+0000 7f62a7829640 1 -- 172.21.15.38:0/3540455587 wait complete. 2024-03-26T06:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a7829640 1 Processor -- start 2024-03-26T06:38:41.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a7829640 1 -- start start 2024-03-26T06:38:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a7829640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a0100440 0x7f62a0199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a7829640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a0100e00 0x7f62a019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a7829640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a0105190 0x7f62a019e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a7829640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a01106b0 con 0x7f62a0100440 2024-03-26T06:38:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a7829640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a0110530 con 0x7f62a0100e00 2024-03-26T06:38:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a7829640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a0110830 con 0x7f62a0105190 2024-03-26T06:38:41.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a55c5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a0100440 0x7f62a0199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a55c5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a0100440 0x7f62a0199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60604/0 (socket says 172.21.15.38:60604) 2024-03-26T06:38:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a55c5640 1 -- 172.21.15.38:0/2573245179 learned_addr learned my addr 172.21.15.38:0/2573245179 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.309+0000 7f62a4dc4640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a0100e00 0x7f62a019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62a5dc6640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a0105190 0x7f62a019e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62a55c5640 1 -- 172.21.15.38:0/2573245179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a0105190 msgr2=0x7f62a019e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a0105190 0x7f62a019e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62a55c5640 1 -- 172.21.15.38:0/2573245179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a0100e00 msgr2=0x7f62a019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a0100e00 0x7f62a019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62a55c5640 1 -- 172.21.15.38:0/2573245179 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a019ed20 con 0x7f62a0100440 2024-03-26T06:38:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a0100440 0x7f62a0199d30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f629c00b4a0 tx=0x7f629c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:41.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62967fc640 1 -- 172.21.15.38:0/2573245179 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c014070 con 0x7f62a0100440 2024-03-26T06:38:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62a7829640 1 -- 172.21.15.38:0/2573245179 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a019f010 con 0x7f62a0100440 2024-03-26T06:38:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62967fc640 1 -- 172.21.15.38:0/2573245179 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f629c010af0 con 0x7f62a0100440 2024-03-26T06:38:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62a7829640 1 -- 172.21.15.38:0/2573245179 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a01a68a0 con 0x7f62a0100440 2024-03-26T06:38:41.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.310+0000 7f62967fc640 1 -- 172.21.15.38:0/2573245179 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c017d10 con 0x7f62a0100440 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62967fc640 1 -- 172.21.15.38:0/2573245179 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f629c017070 con 0x7f62a0100440 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62aa405740 1 Processor -- start 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62aa405740 1 -- start start 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62aa405740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213daa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62aa405740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213db450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62aa405740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213e4550 0x55cc213e6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62aa405740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc2133f5f0 con 0x55cc213dafd0 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62a55c5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213db450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62a55c5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213db450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60606/0 (socket says 172.21.15.38:60606) 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62a55c5640 1 -- 172.21.15.38:0/704785189 learned_addr learned my addr 172.21.15.38:0/704785189 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62aa405740 1 -- 172.21.15.38:0/704785189 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc212e8930 con 0x55cc213e4550 2024-03-26T06:38:41.317 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62aa405740 1 -- 172.21.15.38:0/704785189 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc2133b690 con 0x55cc213dbd40 2024-03-26T06:38:41.318 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62a5dc6640 1 --2- 172.21.15.38:0/704785189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213daa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.314+0000 7f62967fc640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f628c0a2c30 0x7f628c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f62967fc640 1 -- 172.21.15.38:0/2573245179 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f629c0ce2e0 con 0x7f62a0100440 2024-03-26T06:38:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f62a4dc4640 1 --2- 172.21.15.38:0/704785189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213e4550 0x55cc213e6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f62a55c5640 1 -- 172.21.15.38:0/704785189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 msgr2=0x55cc213daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f62a55c5640 1 --2- 172.21.15.38:0/704785189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213daa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f62a55c5640 1 -- 172.21.15.38:0/704785189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213e4550 msgr2=0x55cc213e6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f62a55c5640 1 --2- 172.21.15.38:0/704785189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213e4550 0x55cc213e6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.319 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f62a55c5640 1 -- 172.21.15.38:0/704785189 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc21054210 con 0x55cc213dafd0 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f62a55c5640 1 --2- 172.21.15.38:0/704785189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213db450 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f629c0cba60 tx=0x7f629c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43480fda52f26ea0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f62a5dc6640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f628c0a2c30 0x7f628c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f628b7fe640 1 -- 172.21.15.38:0/704785189 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c01e030 con 0x55cc213dafd0 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f628b7fe640 1 -- 172.21.15.38:0/704785189 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f629c09a790 con 0x55cc213dafd0 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.315+0000 7f628b7fe640 1 -- 172.21.15.38:0/704785189 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c0a2aa0 con 0x55cc213dafd0 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.316+0000 7f62aa405740 1 -- 172.21.15.38:0/704785189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 msgr2=0x55cc213db450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.316+0000 7f62aa405740 1 --2- 172.21.15.38:0/704785189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213db450 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f629c0cba60 tx=0x7f629c0047d0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.316+0000 7f62a5dc6640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f628c0a2c30 0x7f628c0a50f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6298033000 tx=0x7f6298009820 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.316+0000 7f62aa405740 1 -- 172.21.15.38:0/704785189 shutdown_connections 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.316+0000 7f62aa405740 1 --2- 172.21.15.38:0/704785189 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213e4550 0x55cc213e6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.316+0000 7f62aa405740 1 --2- 172.21.15.38:0/704785189 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213db450 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.316+0000 7f62aa405740 1 --2- 172.21.15.38:0/704785189 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213daa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.316+0000 7f62aa405740 1 -- 172.21.15.38:0/704785189 >> 172.21.15.38:0/704785189 conn(0x55cc213a25b0 msgr2=0x55cc213d5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.316+0000 7f62aa405740 1 -- 172.21.15.38:0/704785189 shutdown_connections 2024-03-26T06:38:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.316+0000 7f62aa405740 1 -- 172.21.15.38:0/704785189 wait complete. 2024-03-26T06:38:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62aa405740 1 -- start start 2024-03-26T06:38:41.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62aa405740 1 Processor -- start 2024-03-26T06:38:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62aa405740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213e14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62aa405740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213e1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62aa405740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213e4550 0x55cc213e1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62aa405740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc213e7280 con 0x55cc213dafd0 2024-03-26T06:38:41.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62aa405740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc212fbd90 con 0x55cc213e4550 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62aa405740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc213e7400 con 0x55cc213dbd40 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62a55c5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213e14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62a55c5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213e14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60620/0 (socket says 172.21.15.38:60620) 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62aa405740 10 client.0 _ll_register_callbacks cb 0x55cc21053a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.317+0000 7f62a55c5640 1 -- 172.21.15.38:0/2221798989 learned_addr learned my addr 172.21.15.38:0/2221798989 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f62a4dc4640 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213e1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f62a55c5640 1 -- 172.21.15.38:0/2221798989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 msgr2=0x55cc213e1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213e1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f62a55c5640 1 -- 172.21.15.38:0/2221798989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213e4550 msgr2=0x55cc213e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213e4550 0x55cc213e1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f62a55c5640 1 -- 172.21.15.38:0/2221798989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc21120620 con 0x55cc213dafd0 2024-03-26T06:38:41.324 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f62a4dc4640 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213e1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213e14c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f629c09a490 tx=0x7f629c0cb970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442749bd4e185423 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f6288ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f6288ff9640 1 -- 172.21.15.38:0/2221798989 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c01e070 con 0x55cc213dafd0 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f6288ff9640 1 -- 172.21.15.38:0/2221798989 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f629c017360 con 0x55cc213dafd0 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f62aa405740 1 -- 172.21.15.38:0/2221798989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f629c09a790 con 0x55cc213dafd0 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f6288ff9640 1 -- 172.21.15.38:0/2221798989 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f629c0175e0 con 0x55cc213dafd0 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f62aa405740 10 client.15480 fetch_fsmap learned FSMap version 25 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.318+0000 7f62aa405740 1 -- 172.21.15.38:0/2221798989 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55cc213f3310 con 0x55cc213dafd0 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.319+0000 7f6288ff9640 1 -- 172.21.15.38:0/2221798989 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f629c09a790 con 0x55cc213dafd0 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.319+0000 7f6288ff9640 1 -- 172.21.15.38:0/2221798989 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f629c061ce0 con 0x55cc213dafd0 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.319+0000 7f62aa405740 10 client.15480 fetch_fsmap finished waiting for FSMap version 25 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.319+0000 7f62aa405740 10 client.15480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.319+0000 7f62aa405740 20 client.15480 populate_metadata read hostname 'smithi038' 2024-03-26T06:38:41.325 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.319+0000 7f62aa405740 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55cc213f4660 0x55cc213f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.319+0000 7f62aa405740 4 client.15480 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.319+0000 7f62aa405740 1 -- 172.21.15.38:0/2221798989 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc2134cb30 con 0x55cc213f4660 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.319+0000 7f62a4dc4640 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55cc213f4660 0x55cc213f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.320+0000 7f6288ff9640 10 client.15480 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.320+0000 7f62a4dc4640 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55cc213f4660 0x55cc213f6a50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b961c97c2061bed6 server_cookie=eb85ce428d3d6a72 in_seq=0 out_seq=0 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.320+0000 7f6288ff9640 1 -- 172.21.15.38:0/2221798989 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cc2134cb30 con 0x55cc213f4660 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.320+0000 7f6288ff9640 10 client.15480 handle_command_reply: tid=0 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.320+0000 7f62aa405740 1 client.15480 shutdown 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.320+0000 7f62aa405740 1 -- 172.21.15.38:0/2221798989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 msgr2=0x55cc213e14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.320+0000 7f62aa405740 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213e14c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f629c09a490 tx=0x7f629c0cb970 comp rx=0 tx=0).stop 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.321+0000 7f62aa405740 1 -- 172.21.15.38:0/2221798989 shutdown_connections 2024-03-26T06:38:41.326 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.321+0000 7f62aa405740 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55cc213f4660 0x55cc213f6a50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.321+0000 7f62aa405740 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213e4550 0x55cc213e1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.321+0000 7f62aa405740 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213e1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.321+0000 7f6288ff9640 0 client.15480 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.321+0000 7f62aa405740 1 --2- 172.21.15.38:0/2221798989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dafd0 0x55cc213e14c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.321+0000 7f62aa405740 1 -- 172.21.15.38:0/2221798989 >> 172.21.15.38:0/2221798989 conn(0x55cc213a25b0 msgr2=0x55cc213d7a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.321+0000 7f62aa405740 1 -- 172.21.15.38:0/2221798989 shutdown_connections 2024-03-26T06:38:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.321+0000 7f62aa405740 1 -- 172.21.15.38:0/2221798989 wait complete. 2024-03-26T06:38:41.327 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.321+0000 7f62aa405740 20 client.15480 trim_cache size 0 max 16384 2024-03-26T06:38:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.334+0000 7f62aa405740 1 Processor -- start 2024-03-26T06:38:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.334+0000 7f62aa405740 1 -- start start 2024-03-26T06:38:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.334+0000 7f62aa405740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 0x55cc213e3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.334+0000 7f62aa405740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dbd40 0x55cc213efe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.337 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.334+0000 7f62aa405740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213e4550 0x55cc213de740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.334+0000 7f62aa405740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6284010aa0 con 0x55cc213dbd40 2024-03-26T06:38:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.334+0000 7f62aa405740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628400e640 con 0x55cc213dafd0 2024-03-26T06:38:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.334+0000 7f62aa405740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628400c6e0 con 0x55cc213e4550 2024-03-26T06:38:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f62a4dc4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dbd40 0x55cc213efe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f62a4dc4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dbd40 0x55cc213efe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:60632/0 (socket says 172.21.15.38:60632) 2024-03-26T06:38:41.338 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f62a55c5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 0x55cc213e3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.339 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f62a4dc4640 1 -- 172.21.15.38:0/2921706961 learned_addr learned my addr 172.21.15.38:0/2921706961 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f62a5dc6640 1 --2- 172.21.15.38:0/2921706961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213e4550 0x55cc213de740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f62a55c5640 1 -- 172.21.15.38:0/2921706961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213e4550 msgr2=0x55cc213de740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2921706961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213e4550 0x55cc213de740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f62a55c5640 1 -- 172.21.15.38:0/2921706961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dbd40 msgr2=0x55cc213efe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2921706961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dbd40 0x55cc213efe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f62a55c5640 1 -- 172.21.15.38:0/2921706961 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc213d6e80 con 0x55cc213dafd0 2024-03-26T06:38:41.343 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2921706961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 0x55cc213e3bf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f629c0cbe90 tx=0x7f629c0093e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0e481a7f069e9bc server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f628affd640 1 -- 172.21.15.38:0/2921706961 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c01e070 con 0x55cc213dafd0 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f628affd640 1 -- 172.21.15.38:0/2921706961 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f629c09e8e0 con 0x55cc213dafd0 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.335+0000 7f628affd640 1 -- 172.21.15.38:0/2921706961 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c0aeb70 con 0x55cc213dafd0 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.336+0000 7f62aa405740 1 -- 172.21.15.38:0/2921706961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 msgr2=0x55cc213e3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.336+0000 7f62aa405740 1 --2- 172.21.15.38:0/2921706961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 0x55cc213e3bf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f629c0cbe90 tx=0x7f629c0093e0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.336+0000 7f62aa405740 1 -- 172.21.15.38:0/2921706961 shutdown_connections 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.336+0000 7f62aa405740 1 --2- 172.21.15.38:0/2921706961 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213e4550 0x55cc213de740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.336+0000 7f62aa405740 1 --2- 172.21.15.38:0/2921706961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213dbd40 0x55cc213efe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.336+0000 7f62aa405740 1 --2- 172.21.15.38:0/2921706961 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 0x55cc213e3bf0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.336+0000 7f62aa405740 1 -- 172.21.15.38:0/2921706961 >> 172.21.15.38:0/2921706961 conn(0x55cc213a25b0 msgr2=0x55cc213ef720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.336+0000 7f62aa405740 1 -- 172.21.15.38:0/2921706961 shutdown_connections 2024-03-26T06:38:41.344 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.336+0000 7f62aa405740 1 -- 172.21.15.38:0/2921706961 wait complete. 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62aa405740 1 -- start start 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62aa405740 1 Processor -- start 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62aa405740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 0x55cc213f8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62aa405740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213f7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62aa405740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213e4550 0x55cc213f77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62aa405740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628400cce0 con 0x55cc213e4550 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62aa405740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628400dfe0 con 0x55cc213dafd0 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62aa405740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628400d320 con 0x55cc213dbd40 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62aa405740 10 client.0 _ll_register_callbacks cb 0x7f629c009330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62a55c5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 0x55cc213f8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62a55c5640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 0x55cc213f8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:38696/0 (socket says 172.21.15.38:38696) 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62a55c5640 1 -- 172.21.15.38:0/2774111962 learned_addr learned my addr 172.21.15.38:0/2774111962 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:41.345 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.337+0000 7f62a5dc6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213e4550 0x55cc213f77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f62a55c5640 1 -- 172.21.15.38:0/2774111962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 msgr2=0x55cc213f7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f62a4dc4640 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213f7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213f7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f62a55c5640 1 -- 172.21.15.38:0/2774111962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213e4550 msgr2=0x55cc213f77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213e4550 0x55cc213f77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f62a55c5640 1 -- 172.21.15.38:0/2774111962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc213da2a0 con 0x55cc213dafd0 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f62a55c5640 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 0x55cc213f8e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f629c09a720 tx=0x7f629c0cb940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=303ea89f82671164 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f6288ff9640 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f6288ff9640 1 -- 172.21.15.38:0/2774111962 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c01e070 con 0x55cc213dafd0 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f62aa405740 1 -- 172.21.15.38:0/2774111962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f629c09e8e0 con 0x55cc213dafd0 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f6288ff9640 1 -- 172.21.15.38:0/2774111962 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f629c0aeb70 con 0x55cc213dafd0 2024-03-26T06:38:41.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f6288ff9640 1 -- 172.21.15.38:0/2774111962 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f629c09a790 con 0x55cc213dafd0 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f62aa405740 10 client.25121 fetch_fsmap learned FSMap version 25 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f62aa405740 1 -- 172.21.15.38:0/2774111962 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55cc213da2a0 con 0x55cc213dafd0 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.338+0000 7f6288ff9640 1 -- 172.21.15.38:0/2774111962 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f629c09e8e0 con 0x55cc213dafd0 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.339+0000 7f6288ff9640 1 -- 172.21.15.38:0/2774111962 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f629c09bbe0 con 0x55cc213dafd0 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.339+0000 7f62aa405740 10 client.25121 fetch_fsmap finished waiting for FSMap version 25 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.339+0000 7f62aa405740 10 client.25121 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.339+0000 7f62aa405740 20 client.25121 populate_metadata read hostname 'smithi038' 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.339+0000 7f62aa405740 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55cc2144b8e0 0x55cc2144dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.339+0000 7f62aa405740 4 client.25121 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.339+0000 7f62aa405740 1 -- 172.21.15.38:0/2774111962 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cc21346e80 con 0x55cc2144b8e0 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.339+0000 7f62a4dc4640 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55cc2144b8e0 0x55cc2144dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.339+0000 7f6288ff9640 10 client.25121 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:41.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.339+0000 7f62a4dc4640 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55cc2144b8e0 0x55cc2144dcd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87e75cc92d5e040c server_cookie=72728d14334fc16b in_seq=0 out_seq=0 2024-03-26T06:38:41.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f6288ff9640 1 -- 172.21.15.38:0/2774111962 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cc21346e80 con 0x55cc2144b8e0 2024-03-26T06:38:41.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f6288ff9640 10 client.25121 handle_command_reply: tid=0 2024-03-26T06:38:41.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f62aa405740 1 client.25121 shutdown 2024-03-26T06:38:41.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f62aa405740 1 -- 172.21.15.38:0/2774111962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 msgr2=0x55cc213f8e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:41.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f62aa405740 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 0x55cc213f8e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f629c09a720 tx=0x7f629c0cb940 comp rx=0 tx=0).stop 2024-03-26T06:38:41.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f62aa405740 1 -- 172.21.15.38:0/2774111962 shutdown_connections 2024-03-26T06:38:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f62aa405740 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55cc2144b8e0 0x55cc2144dcd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f62aa405740 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55cc213e4550 0x55cc213f77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f6288ff9640 0 client.25121 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f62aa405740 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55cc213dbd40 0x55cc213f7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f62aa405740 1 --2- 172.21.15.38:0/2774111962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55cc213dafd0 0x55cc213f8e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.343+0000 7f62aa405740 1 -- 172.21.15.38:0/2774111962 >> 172.21.15.38:0/2774111962 conn(0x55cc213a25b0 msgr2=0x55cc213effb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62aa405740 1 -- 172.21.15.38:0/2774111962 shutdown_connections 2024-03-26T06:38:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62aa405740 1 -- 172.21.15.38:0/2774111962 wait complete. 2024-03-26T06:38:41.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62aa405740 20 client.25121 trim_cache size 0 max 16384 2024-03-26T06:38:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:38:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:38:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:38:41.351 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:38:41.352 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 -- 172.21.15.38:0/2573245179 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f628c0a2c30 msgr2=0x7f628c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:41.352 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f628c0a2c30 0x7f628c0a50f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6298033000 tx=0x7f6298009820 comp rx=0 tx=0).stop 2024-03-26T06:38:41.352 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 -- 172.21.15.38:0/2573245179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a0100440 msgr2=0x7f62a0199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:41.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a0100440 0x7f62a0199d30 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f629c00b4a0 tx=0x7f629c00b960 comp rx=0 tx=0).stop 2024-03-26T06:38:41.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 -- 172.21.15.38:0/2573245179 shutdown_connections 2024-03-26T06:38:41.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f628c0a2c30 0x7f628c0a50f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f62a0105190 0x7f62a019e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62a0100e00 0x7f62a019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.353 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 --2- 172.21.15.38:0/2573245179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f62a0100440 0x7f62a0199d30 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 -- 172.21.15.38:0/2573245179 >> 172.21.15.38:0/2573245179 conn(0x7f62a0077040 msgr2=0x7f62a0068970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 -- 172.21.15.38:0/2573245179 shutdown_connections 2024-03-26T06:38:41.354 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:41.344+0000 7f62a7829640 1 -- 172.21.15.38:0/2573245179 wait complete. 2024-03-26T06:38:42.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:42 smithi170 ceph-mon[36912]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:42.447 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:42 smithi121 ceph-mon[36349]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:42.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:42 smithi038 ceph-mon[32265]: pgmap v38: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:44.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:44 smithi038 ceph-mon[32265]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:38:44.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:44 smithi121 ceph-mon[36349]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:38:44.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:44 smithi170 ceph-mon[36912]: pgmap v39: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:38:46.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:46 smithi038 ceph-mon[32265]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:46.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:46 smithi121 ceph-mon[36349]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:46.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:46 smithi170 ceph-mon[36912]: pgmap v40: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:48.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:48 smithi038 ceph-mon[32265]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:48.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:48 smithi121 ceph-mon[36349]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:48.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:48 smithi170 ceph-mon[36912]: pgmap v41: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:50.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:50 smithi170 ceph-mon[36912]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:50.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:50 smithi038 ceph-mon[32265]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:50.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:50 smithi121 ceph-mon[36349]: pgmap v42: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:52 smithi038 ceph-mon[32265]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:52 smithi121 ceph-mon[36349]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:52.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:52 smithi170 ceph-mon[36912]: pgmap v43: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:53.592 INFO:tasks.workunit.client.0.smithi038.stderr:Updating files: 93% (11502/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-26T06:38:54.176 INFO:tasks.workunit.client.0.smithi038.stderr:Note: switching to '7e98f5f6b535defc7fa715b456904ffc0500aa9f'. 2024-03-26T06:38:54.176 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr:state without impacting any branches by switching back to a branch. 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr: git switch -c 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr:Or undo this operation with: 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr: git switch - 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-03-26T06:38:54.177 INFO:tasks.workunit.client.0.smithi038.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-26T06:38:54.178 INFO:tasks.workunit.client.0.smithi038.stderr: 2024-03-26T06:38:54.178 INFO:tasks.workunit.client.0.smithi038.stderr:HEAD is now at 7e98f5f6b53 DNM: just to trigger builds again 2024-03-26T06:38:54.180 DEBUG:teuthology.orchestra.run.smithi038:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-26T06:38:54.248 INFO:tasks.workunit.client.0.smithi038.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-26T06:38:54.250 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T06:38:54.250 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-26T06:38:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.432+0000 7f9ee7513640 1 -- 172.21.15.38:0/2242727666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ee0101ec0 msgr2=0x7f9ee0105f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.432+0000 7f9ee7513640 1 --2- 172.21.15.38:0/2242727666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ee0101ec0 0x7f9ee0105f00 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0009900 tx=0x7f9ed002bc30 comp rx=0 tx=0).stop 2024-03-26T06:38:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.432+0000 7f9ee7513640 1 -- 172.21.15.38:0/2242727666 shutdown_connections 2024-03-26T06:38:54.437 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.432+0000 7f9ee7513640 1 --2- 172.21.15.38:0/2242727666 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee01065e0 0x7f9ee010e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.432+0000 7f9ee7513640 1 --2- 172.21.15.38:0/2242727666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ee0101ec0 0x7f9ee0105f00 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.432+0000 7f9ee7513640 1 --2- 172.21.15.38:0/2242727666 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ee0101510 0x7f9ee01018f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.432+0000 7f9ee7513640 1 -- 172.21.15.38:0/2242727666 >> 172.21.15.38:0/2242727666 conn(0x7f9ee00fcdb0 msgr2=0x7f9ee00ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.432+0000 7f9ee7513640 1 -- 172.21.15.38:0/2242727666 shutdown_connections 2024-03-26T06:38:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.432+0000 7f9ee7513640 1 -- 172.21.15.38:0/2242727666 wait complete. 2024-03-26T06:38:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee7513640 1 Processor -- start 2024-03-26T06:38:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee7513640 1 -- start start 2024-03-26T06:38:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee7513640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0101510 0x7f9ee019d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee7513640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ee01065e0 0x7f9ee019dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee7513640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ee01979d0 0x7f9ee0197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee7513640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee0111670 con 0x7f9ee01065e0 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee7513640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee01114f0 con 0x7f9ee01979d0 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee7513640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee01117f0 con 0x7f9ee0101510 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee4aae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ee01065e0 0x7f9ee019dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee4aae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ee01065e0 0x7f9ee019dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47260/0 (socket says 172.21.15.38:47260) 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee4aae640 1 -- 172.21.15.38:0/47722552 learned_addr learned my addr 172.21.15.38:0/47722552 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee5ab0640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ee01979d0 0x7f9ee0197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee52af640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0101510 0x7f9ee019d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee4aae640 1 -- 172.21.15.38:0/47722552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0101510 msgr2=0x7f9ee019d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0101510 0x7f9ee019d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee4aae640 1 -- 172.21.15.38:0/47722552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ee01979d0 msgr2=0x7f9ee0197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.433+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ee01979d0 0x7f9ee0197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.434+0000 7f9ee4aae640 1 -- 172.21.15.38:0/47722552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee01984f0 con 0x7f9ee01065e0 2024-03-26T06:38:54.439 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.434+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ee01065e0 0x7f9ee019dd10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0036040 tx=0x7f9ed002ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.434+0000 7f9ed67fc640 1 -- 172.21.15.38:0/47722552 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0041070 con 0x7f9ee01065e0 2024-03-26T06:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.434+0000 7f9ee7513640 1 -- 172.21.15.38:0/47722552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee0198780 con 0x7f9ee01065e0 2024-03-26T06:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.434+0000 7f9ed67fc640 1 -- 172.21.15.38:0/47722552 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ed00054e0 con 0x7f9ee01065e0 2024-03-26T06:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.434+0000 7f9ed67fc640 1 -- 172.21.15.38:0/47722552 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed0034440 con 0x7f9ee01065e0 2024-03-26T06:38:54.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.434+0000 7f9ee7513640 1 -- 172.21.15.38:0/47722552 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee01a4930 con 0x7f9ee01065e0 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.436+0000 7f9ed67fc640 1 -- 172.21.15.38:0/47722552 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f9ed0044070 con 0x7f9ee01065e0 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9eea1c5740 1 Processor -- start 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9eea1c5740 1 -- start start 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ed67fc640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f9ecc0a2b30 0x7f9ecc0a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 0x556e5cec2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec2fd0 0x556e5cec3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ed67fc640 1 -- 172.21.15.38:0/47722552 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f9ed00f0c40 con 0x7f9ee01065e0 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cecc550 0x556e5cece940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e5ce275f0 con 0x556e5cec2fd0 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e5cdd0930 con 0x556e5cec3d40 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ee5ab0640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f9ecc0a2b30 0x7f9ecc0a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.447 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e5ce23690 con 0x556e5cecc550 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ee4aae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec2fd0 0x556e5cec3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ee4aae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec2fd0 0x556e5cec3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47264/0 (socket says 172.21.15.38:47264) 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ee4aae640 1 -- 172.21.15.38:0/2053254787 learned_addr learned my addr 172.21.15.38:0/2053254787 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ee52af640 1 --2- 172.21.15.38:0/2053254787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 0x556e5cec2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ee4aae640 1 -- 172.21.15.38:0/2053254787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cecc550 msgr2=0x556e5cece940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/2053254787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cecc550 0x556e5cece940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ee4aae640 1 -- 172.21.15.38:0/2053254787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 msgr2=0x556e5cec2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/2053254787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 0x556e5cec2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ee5ab0640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f9ecc0a2b30 0x7f9ecc0a4ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8005fd0 tx=0x7f9ed8014b50 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.437+0000 7f9ee4aae640 1 -- 172.21.15.38:0/2053254787 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e5cb3c210 con 0x556e5cec2fd0 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/2053254787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec2fd0 0x556e5cec3450 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00bcee0 tx=0x7f9ed0004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c695dce219da5e server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9ecb7fe640 1 -- 172.21.15.38:0/2053254787 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed004b070 con 0x556e5cec2fd0 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9ecb7fe640 1 -- 172.21.15.38:0/2053254787 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ed00c1120 con 0x556e5cec2fd0 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9ecb7fe640 1 -- 172.21.15.38:0/2053254787 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed00c58f0 con 0x556e5cec2fd0 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9eea1c5740 1 -- 172.21.15.38:0/2053254787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec2fd0 msgr2=0x556e5cec3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/2053254787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec2fd0 0x556e5cec3450 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00bcee0 tx=0x7f9ed0004610 comp rx=0 tx=0).stop 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9eea1c5740 1 -- 172.21.15.38:0/2053254787 shutdown_connections 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/2053254787 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cecc550 0x556e5cece940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/2053254787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec2fd0 0x556e5cec3450 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/2053254787 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 0x556e5cec2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9eea1c5740 1 -- 172.21.15.38:0/2053254787 >> 172.21.15.38:0/2053254787 conn(0x556e5ce8a5b0 msgr2=0x556e5cebdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9eea1c5740 1 -- 172.21.15.38:0/2053254787 shutdown_connections 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.438+0000 7f9eea1c5740 1 -- 172.21.15.38:0/2053254787 wait complete. 2024-03-26T06:38:54.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9eea1c5740 1 -- start start 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9eea1c5740 1 Processor -- start 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec2fd0 0x556e5cec9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec3d40 0x556e5cec9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cecc550 0x556e5ceca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e5ce275f0 con 0x556e5cec3d40 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e5cde3d90 con 0x556e5cec2fd0 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e5cecf040 con 0x556e5cecc550 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9eea1c5740 10 client.0 _ll_register_callbacks cb 0x556e5cb3ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9ee4aae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec3d40 0x556e5cec9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9ee4aae640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec3d40 0x556e5cec9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47266/0 (socket says 172.21.15.38:47266) 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9ee4aae640 1 -- 172.21.15.38:0/1219707962 learned_addr learned my addr 172.21.15.38:0/1219707962 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.439+0000 7f9ee52af640 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec2fd0 0x556e5cec9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ee5ab0640 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cecc550 0x556e5ceca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ee4aae640 1 -- 172.21.15.38:0/1219707962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cecc550 msgr2=0x556e5ceca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cecc550 0x556e5ceca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ee4aae640 1 -- 172.21.15.38:0/1219707962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec2fd0 msgr2=0x556e5cec9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec2fd0 0x556e5cec9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ee4aae640 1 -- 172.21.15.38:0/1219707962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e5cc08620 con 0x556e5cec3d40 2024-03-26T06:38:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ee52af640 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec2fd0 0x556e5cec9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:38:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec3d40 0x556e5cec9b40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00ee980 tx=0x7f9ed00ee280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33755efdef8a2781 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ec8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:38:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1219707962 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed00d5020 con 0x556e5cec3d40 2024-03-26T06:38:54.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1219707962 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f9ed00c5660 con 0x556e5cec3d40 2024-03-26T06:38:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1219707962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ed00c1120 con 0x556e5cec3d40 2024-03-26T06:38:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1219707962 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ed009f030 con 0x556e5cec3d40 2024-03-26T06:38:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9eea1c5740 10 client.15498 fetch_fsmap learned FSMap version 25 2024-03-26T06:38:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1219707962 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556e5cedb230 con 0x556e5cec3d40 2024-03-26T06:38:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1219707962 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ed00c1120 con 0x556e5cec3d40 2024-03-26T06:38:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.440+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1219707962 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9ed00a6370 con 0x556e5cec3d40 2024-03-26T06:38:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.441+0000 7f9eea1c5740 10 client.15498 fetch_fsmap finished waiting for FSMap version 25 2024-03-26T06:38:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.441+0000 7f9eea1c5740 10 client.15498 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:38:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.441+0000 7f9eea1c5740 20 client.15498 populate_metadata read hostname 'smithi038' 2024-03-26T06:38:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.441+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556e5cedc580 0x556e5cede970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.452 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.441+0000 7f9eea1c5740 4 client.15498 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:38:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.441+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1219707962 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556e5ce34b30 con 0x556e5cedc580 2024-03-26T06:38:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.441+0000 7f9ee52af640 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556e5cedc580 0x556e5cede970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.441+0000 7f9ee52af640 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556e5cedc580 0x556e5cede970 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e20fe19a790d002e server_cookie=9b745a30f9cb902a in_seq=0 out_seq=0 2024-03-26T06:38:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.441+0000 7f9ec8ff9640 10 client.15498 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1219707962 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556e5ce34b30 con 0x556e5cedc580 2024-03-26T06:38:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9ec8ff9640 10 client.15498 handle_command_reply: tid=0 2024-03-26T06:38:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 1 client.15498 shutdown 2024-03-26T06:38:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1219707962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec3d40 msgr2=0x556e5cec9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec3d40 0x556e5cec9b40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00ee980 tx=0x7f9ed00ee280 comp rx=0 tx=0).stop 2024-03-26T06:38:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1219707962 shutdown_connections 2024-03-26T06:38:54.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556e5cedc580 0x556e5cede970 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cecc550 0x556e5ceca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9ec8ff9640 0 client.15498 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5cec3d40 0x556e5cec9b40 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1219707962 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec2fd0 0x556e5cec9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1219707962 >> 172.21.15.38:0/1219707962 conn(0x556e5ce8a5b0 msgr2=0x556e5cebfa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1219707962 shutdown_connections 2024-03-26T06:38:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1219707962 wait complete. 2024-03-26T06:38:54.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.442+0000 7f9eea1c5740 20 client.15498 trim_cache size 0 max 16384 2024-03-26T06:38:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.455+0000 7f9eea1c5740 1 Processor -- start 2024-03-26T06:38:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.455+0000 7f9eea1c5740 1 -- start start 2024-03-26T06:38:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cec2fd0 0x556e5ced32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 0x556e5ced37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee01ad0b0 con 0x556e5ce8a5b0 2024-03-26T06:38:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee01aac50 con 0x556e5cec3d40 2024-03-26T06:38:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee01a8d10 con 0x556e5cec2fd0 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ee52af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ee52af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47278/0 (socket says 172.21.15.38:47278) 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ee52af640 1 -- 172.21.15.38:0/2313323965 learned_addr learned my addr 172.21.15.38:0/2313323965 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ee52af640 1 -- 172.21.15.38:0/2313323965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cec2fd0 msgr2=0x556e5ced32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ee52af640 1 --2- 172.21.15.38:0/2313323965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cec2fd0 0x556e5ced32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ee52af640 1 -- 172.21.15.38:0/2313323965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 msgr2=0x556e5ced37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ee52af640 1 --2- 172.21.15.38:0/2313323965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 0x556e5ced37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ee52af640 1 -- 172.21.15.38:0/2313323965 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e5cec22a0 con 0x556e5ce8a5b0 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ee52af640 1 --2- 172.21.15.38:0/2313323965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecbd30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x556e5cedf120 tx=0x7f9edc013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dca197b8776f700 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ecaffd640 1 -- 172.21.15.38:0/2313323965 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc004030 con 0x556e5ce8a5b0 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ecaffd640 1 -- 172.21.15.38:0/2313323965 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9edc01c890 con 0x556e5ce8a5b0 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.456+0000 7f9ecaffd640 1 -- 172.21.15.38:0/2313323965 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc003bb0 con 0x556e5ce8a5b0 2024-03-26T06:38:54.461 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.457+0000 7f9eea1c5740 1 -- 172.21.15.38:0/2313323965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 msgr2=0x556e5cecbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.457+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/2313323965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecbd30 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x556e5cedf120 tx=0x7f9edc013830 comp rx=0 tx=0).stop 2024-03-26T06:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.457+0000 7f9eea1c5740 1 -- 172.21.15.38:0/2313323965 shutdown_connections 2024-03-26T06:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.457+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/2313323965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 0x556e5ced37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.457+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/2313323965 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cec2fd0 0x556e5ced32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.457+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/2313323965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecbd30 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.457+0000 7f9eea1c5740 1 -- 172.21.15.38:0/2313323965 >> 172.21.15.38:0/2313323965 conn(0x556e5cedece0 msgr2=0x556e5cebd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.457+0000 7f9eea1c5740 1 -- 172.21.15.38:0/2313323965 shutdown_connections 2024-03-26T06:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.457+0000 7f9eea1c5740 1 -- 172.21.15.38:0/2313323965 wait complete. 2024-03-26T06:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.457+0000 7f9eea1c5740 1 -- start start 2024-03-26T06:38:54.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.457+0000 7f9eea1c5740 1 Processor -- start 2024-03-26T06:38:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cec2fd0 0x556e5ced01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9eea1c5740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 0x556e5ced0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee01a9310 con 0x556e5ce8a5b0 2024-03-26T06:38:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee01aa5f0 con 0x556e5cec3d40 2024-03-26T06:38:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9eea1c5740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee01a9930 con 0x556e5cec2fd0 2024-03-26T06:38:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9eea1c5740 10 client.0 _ll_register_callbacks cb 0x7f9ed0005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:38:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee52af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee52af640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecfcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47280/0 (socket says 172.21.15.38:47280) 2024-03-26T06:38:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee52af640 1 -- 172.21.15.38:0/1296346167 learned_addr learned my addr 172.21.15.38:0/1296346167 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:38:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee5ab0640 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 0x556e5ced0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cec2fd0 0x556e5ced01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee52af640 1 -- 172.21.15.38:0/1296346167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cec2fd0 msgr2=0x556e5ced01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee52af640 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cec2fd0 0x556e5ced01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee52af640 1 -- 172.21.15.38:0/1296346167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 msgr2=0x556e5ced0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee52af640 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 0x556e5ced0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee52af640 1 -- 172.21.15.38:0/1296346167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e5cf32920 con 0x556e5ce8a5b0 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cec2fd0 0x556e5ced01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ee52af640 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecfcb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9edc000c00 tx=0x7f9edc013110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713046746c043ae server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.458+0000 7f9ec8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1296346167 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc011440 con 0x556e5ce8a5b0 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1296346167 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f9edc0115e0 con 0x556e5ce8a5b0 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1296346167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9edc01c890 con 0x556e5ce8a5b0 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1296346167 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9edc0117e0 con 0x556e5ce8a5b0 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9eea1c5740 10 client.15504 fetch_fsmap learned FSMap version 25 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1296346167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556e5cf33c10 con 0x556e5ce8a5b0 2024-03-26T06:38:54.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1296346167 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9edc01c890 con 0x556e5ce8a5b0 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1296346167 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9edc025a90 con 0x556e5ce8a5b0 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9eea1c5740 10 client.15504 fetch_fsmap finished waiting for FSMap version 25 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9eea1c5740 10 client.15504 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9eea1c5740 20 client.15504 populate_metadata read hostname 'smithi038' 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556e5cecc550 0x556e5cf362c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9eea1c5740 4 client.15504 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1296346167 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556e5ce2ee80 con 0x556e5cecc550 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.459+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556e5cecc550 0x556e5cf362c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.460+0000 7f9ec8ff9640 10 client.15504 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.460+0000 7f9ee4aae640 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556e5cecc550 0x556e5cf362c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e97bb8aa4d4620c5 server_cookie=e6700776f037cbe8 in_seq=0 out_seq=0 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9ec8ff9640 1 -- 172.21.15.38:0/1296346167 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556e5ce2ee80 con 0x556e5cecc550 2024-03-26T06:38:54.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9ec8ff9640 10 client.15504 handle_command_reply: tid=0 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9eea1c5740 1 client.15504 shutdown 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1296346167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 msgr2=0x556e5cecfcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecfcb0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9edc000c00 tx=0x7f9edc013110 comp rx=0 tx=0).stop 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1296346167 shutdown_connections 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x556e5cecc550 0x556e5cf362c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x556e5cec3d40 0x556e5ced0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9ec8ff9640 0 client.15504 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x556e5cec2fd0 0x556e5ced01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9eea1c5740 1 --2- 172.21.15.38:0/1296346167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556e5ce8a5b0 0x556e5cecfcb0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.463+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1296346167 >> 172.21.15.38:0/1296346167 conn(0x556e5cedece0 msgr2=0x556e5cec7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1296346167 shutdown_connections 2024-03-26T06:38:54.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9eea1c5740 1 -- 172.21.15.38:0/1296346167 wait complete. 2024-03-26T06:38:54.471 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9eea1c5740 20 client.15504 trim_cache size 0 max 16384 2024-03-26T06:38:54.471 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:38:54.471 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:38:54.471 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:38:54.471 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:38:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 -- 172.21.15.38:0/47722552 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f9ecc0a2b30 msgr2=0x7f9ecc0a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f9ecc0a2b30 0x7f9ecc0a4ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8005fd0 tx=0x7f9ed8014b50 comp rx=0 tx=0).stop 2024-03-26T06:38:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 -- 172.21.15.38:0/47722552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ee01065e0 msgr2=0x7f9ee019dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:38:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ee01065e0 0x7f9ee019dd10 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0036040 tx=0x7f9ed002ce20 comp rx=0 tx=0).stop 2024-03-26T06:38:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 -- 172.21.15.38:0/47722552 shutdown_connections 2024-03-26T06:38:54.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f9ecc0a2b30 0x7f9ecc0a4ff0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9ee01979d0 0x7f9ee0197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9ee01065e0 0x7f9ee019dd10 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 --2- 172.21.15.38:0/47722552 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f9ee0101510 0x7f9ee019d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:38:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 -- 172.21.15.38:0/47722552 >> 172.21.15.38:0/47722552 conn(0x7f9ee00fcdb0 msgr2=0x7f9ee00ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:38:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 -- 172.21.15.38:0/47722552 shutdown_connections 2024-03-26T06:38:54.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:38:54.464+0000 7f9ee7513640 1 -- 172.21.15.38:0/47722552 wait complete. 2024-03-26T06:38:54.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:54 smithi038 ceph-mon[32265]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:54.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:54 smithi121 ceph-mon[36349]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:54.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:54 smithi170 ceph-mon[36912]: pgmap v44: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-03-26T06:38:55.032 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-26T06:38:55.097 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-26T06:38:55.137 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T06:38:55.138 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T06:38:55.138 INFO:tasks.workunit.client.0.smithi038.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-26T06:38:55.177 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T06:38:55.181 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:38:55.181 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-26T06:38:55.237 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-03-26T06:38:55.238 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-03-26T06:38:55.238 DEBUG:teuthology.orchestra.run.smithi038:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7e98f5f6b535defc7fa715b456904ffc0500aa9f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-26T06:38:55.298 INFO:tasks.workunit.client.0.smithi038.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-26T06:38:55.299 INFO:tasks.workunit.client.0.smithi038.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-03-26T06:38:55.299 INFO:tasks.workunit.client.0.smithi038.stderr:+ trap -- retry EXIT 2024-03-26T06:38:55.299 INFO:tasks.workunit.client.0.smithi038.stderr:+ rm -rf ffsb 2024-03-26T06:38:55.299 INFO:tasks.workunit.client.0.smithi038.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-03-26T06:38:55.302 INFO:tasks.workunit.client.0.smithi038.stderr:Cloning into 'ffsb'... 2024-03-26T06:38:56.274 INFO:tasks.workunit.client.0.smithi038.stderr:+ trap - EXIT 2024-03-26T06:38:56.275 INFO:tasks.workunit.client.0.smithi038.stderr:+ cd ffsb 2024-03-26T06:38:56.275 INFO:tasks.workunit.client.0.smithi038.stderr:+ ./configure 2024-03-26T06:38:56.387 INFO:tasks.workunit.client.0.smithi038.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-26T06:38:56.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:56 smithi038 ceph-mon[32265]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:56.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:56 smithi121 ceph-mon[36349]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:56.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:56 smithi170 ceph-mon[36912]: pgmap v45: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-26T06:38:57.394 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether build environment is sane... yes 2024-03-26T06:38:57.403 INFO:tasks.workunit.client.0.smithi038.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-26T06:38:57.403 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gawk... gawk 2024-03-26T06:38:57.424 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether make sets $(MAKE)... yes 2024-03-26T06:38:57.460 INFO:tasks.workunit.client.0.smithi038.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-26T06:38:57.461 INFO:tasks.workunit.client.0.smithi038.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-26T06:38:57.469 INFO:tasks.workunit.client.0.smithi038.stdout:checking for style of include used by make... GNU 2024-03-26T06:38:57.475 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc... gcc 2024-03-26T06:38:57.626 INFO:tasks.workunit.client.0.smithi038.stdout:checking for C compiler default output file name... a.out 2024-03-26T06:38:57.627 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether the C compiler works... yes 2024-03-26T06:38:57.630 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are cross compiling... no 2024-03-26T06:38:57.669 INFO:tasks.workunit.client.0.smithi038.stdout:checking for suffix of executables... 2024-03-26T06:38:57.704 INFO:tasks.workunit.client.0.smithi038.stdout:checking for suffix of object files... o 2024-03-26T06:38:57.733 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are using the GNU C compiler... yes 2024-03-26T06:38:57.762 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether gcc accepts -g... yes 2024-03-26T06:38:57.830 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-26T06:38:57.932 INFO:tasks.workunit.client.0.smithi038.stdout:checking dependency style of gcc... gcc3 2024-03-26T06:38:57.989 INFO:tasks.workunit.client.0.smithi038.stdout:checking how to run the C preprocessor... gcc -E 2024-03-26T06:38:58.027 INFO:tasks.workunit.client.0.smithi038.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-26T06:38:58.028 INFO:tasks.workunit.client.0.smithi038.stdout:checking for egrep... /usr/bin/grep -E 2024-03-26T06:38:58.035 INFO:tasks.workunit.client.0.smithi038.stdout:checking for AIX... no 2024-03-26T06:38:58.054 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc... (cached) gcc 2024-03-26T06:38:58.063 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-26T06:38:58.063 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-26T06:38:58.063 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-26T06:38:58.063 INFO:tasks.workunit.client.0.smithi038.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-26T06:38:58.118 INFO:tasks.workunit.client.0.smithi038.stdout:checking for main in -lm... yes 2024-03-26T06:38:58.162 INFO:tasks.workunit.client.0.smithi038.stdout:checking for main in -lpthread... yes 2024-03-26T06:38:58.320 INFO:tasks.workunit.client.0.smithi038.stdout:checking for ANSI C header files... yes 2024-03-26T06:38:58.382 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-26T06:38:58.423 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/types.h... yes 2024-03-26T06:38:58.467 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/stat.h... yes 2024-03-26T06:38:58.510 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdlib.h... yes 2024-03-26T06:38:58.554 INFO:tasks.workunit.client.0.smithi038.stdout:checking for string.h... yes 2024-03-26T06:38:58.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:58 smithi038 ceph-mon[32265]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:38:58.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:38:58 smithi038 ceph-mon[32265]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:38:58.605 INFO:tasks.workunit.client.0.smithi038.stdout:checking for memory.h... yes 2024-03-26T06:38:58.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:58 smithi121 ceph-mon[36349]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:38:58.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:38:58 smithi121 ceph-mon[36349]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:38:58.648 INFO:tasks.workunit.client.0.smithi038.stdout:checking for strings.h... yes 2024-03-26T06:38:58.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:58 smithi170 ceph-mon[36912]: pgmap v46: 97 pgs: 97 active+clean; 1.4 MiB data, 365 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:38:58.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:38:58 smithi170 ceph-mon[36912]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:38:58.691 INFO:tasks.workunit.client.0.smithi038.stdout:checking for inttypes.h... yes 2024-03-26T06:38:58.735 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdint.h... yes 2024-03-26T06:38:58.781 INFO:tasks.workunit.client.0.smithi038.stdout:checking for unistd.h... yes 2024-03-26T06:38:58.839 INFO:tasks.workunit.client.0.smithi038.stdout:checking pthread.h usability... yes 2024-03-26T06:38:58.857 INFO:tasks.workunit.client.0.smithi038.stdout:checking pthread.h presence... yes 2024-03-26T06:38:58.857 INFO:tasks.workunit.client.0.smithi038.stdout:checking for pthread.h... yes 2024-03-26T06:38:58.904 INFO:tasks.workunit.client.0.smithi038.stdout:checking fcntl.h usability... yes 2024-03-26T06:38:58.920 INFO:tasks.workunit.client.0.smithi038.stdout:checking fcntl.h presence... yes 2024-03-26T06:38:58.920 INFO:tasks.workunit.client.0.smithi038.stdout:checking for fcntl.h... yes 2024-03-26T06:38:58.971 INFO:tasks.workunit.client.0.smithi038.stdout:checking limits.h usability... yes 2024-03-26T06:38:58.987 INFO:tasks.workunit.client.0.smithi038.stdout:checking limits.h presence... yes 2024-03-26T06:38:58.987 INFO:tasks.workunit.client.0.smithi038.stdout:checking for limits.h... yes 2024-03-26T06:38:59.005 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdint.h... (cached) yes 2024-03-26T06:38:59.059 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/time.h usability... yes 2024-03-26T06:38:59.075 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/time.h presence... yes 2024-03-26T06:38:59.075 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/time.h... yes 2024-03-26T06:38:59.094 INFO:tasks.workunit.client.0.smithi038.stdout:checking for unistd.h... (cached) yes 2024-03-26T06:38:59.131 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/vfs.h usability... yes 2024-03-26T06:38:59.147 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/vfs.h presence... yes 2024-03-26T06:38:59.147 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/vfs.h... yes 2024-03-26T06:38:59.190 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/limits.h usability... no 2024-03-26T06:38:59.206 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/limits.h presence... no 2024-03-26T06:38:59.207 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/limits.h... no 2024-03-26T06:38:59.234 INFO:tasks.workunit.client.0.smithi038.stdout:checking for an ANSI C-conforming const... yes 2024-03-26T06:38:59.262 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-26T06:38:59.263 INFO:tasks.workunit.client.0.smithi038.stdout:checking for function prototypes... yes 2024-03-26T06:38:59.286 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether setvbuf arguments are reversed... no 2024-03-26T06:38:59.354 INFO:tasks.workunit.client.0.smithi038.stdout:checking for system... yes 2024-03-26T06:38:59.411 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gettimeofday... yes 2024-03-26T06:38:59.468 INFO:tasks.workunit.client.0.smithi038.stdout:checking for mkdir... yes 2024-03-26T06:38:59.524 INFO:tasks.workunit.client.0.smithi038.stdout:checking for strerror... yes 2024-03-26T06:38:59.581 INFO:tasks.workunit.client.0.smithi038.stdout:checking for open64... yes 2024-03-26T06:38:59.636 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stat64... yes 2024-03-26T06:38:59.694 INFO:tasks.workunit.client.0.smithi038.stdout:checking for fseeko64... yes 2024-03-26T06:38:59.753 INFO:tasks.workunit.client.0.smithi038.stdout:checking for lrand48_r... yes 2024-03-26T06:38:59.811 INFO:tasks.workunit.client.0.smithi038.stdout:checking for srand48_r... yes 2024-03-26T06:38:59.836 INFO:tasks.workunit.client.0.smithi038.stdout:configure: creating ./config.status 2024-03-26T06:38:59.927 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: creating Makefile 2024-03-26T06:38:59.938 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: creating config.h 2024-03-26T06:38:59.951 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: executing depfiles commands 2024-03-26T06:38:59.980 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: executing default commands 2024-03-26T06:39:00.027 INFO:tasks.workunit.client.0.smithi038.stderr:+ make 2024-03-26T06:39:00.034 INFO:tasks.workunit.client.0.smithi038.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-03-26T06:39:00.035 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-03-26T06:39:00.036 INFO:tasks.workunit.client.0.smithi038.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-03-26T06:39:00.036 INFO:tasks.workunit.client.0.smithi038.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-03-26T06:39:00.036 INFO:tasks.workunit.client.0.smithi038.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-03-26T06:39:00.036 INFO:tasks.workunit.client.0.smithi038.stderr: any GNU archive site. 2024-03-26T06:39:00.038 INFO:tasks.workunit.client.0.smithi038.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-03-26T06:39:00.039 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-03-26T06:39:00.040 INFO:tasks.workunit.client.0.smithi038.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-03-26T06:39:00.040 INFO:tasks.workunit.client.0.smithi038.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-03-26T06:39:00.040 INFO:tasks.workunit.client.0.smithi038.stderr: You might want to install the `Automake' and `Perl' packages. 2024-03-26T06:39:00.040 INFO:tasks.workunit.client.0.smithi038.stderr: Grab them from any GNU archive site. 2024-03-26T06:39:00.101 INFO:tasks.workunit.client.0.smithi038.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-03-26T06:39:00.425 INFO:tasks.workunit.client.0.smithi038.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-03-26T06:39:00.425 INFO:tasks.workunit.client.0.smithi038.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-03-26T06:39:00.425 INFO:tasks.workunit.client.0.smithi038.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-03-26T06:39:00.425 INFO:tasks.workunit.client.0.smithi038.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-03-26T06:39:00.546 INFO:tasks.workunit.client.0.smithi038.stdout:/bin/sh ./config.status --recheck 2024-03-26T06:39:00.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:00 smithi038 ceph-mon[32265]: pgmap v47: 97 pgs: 97 active+clean; 5.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 327 KiB/s wr, 28 op/s 2024-03-26T06:39:00.595 INFO:tasks.workunit.client.0.smithi038.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-03-26T06:39:00.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:00 smithi121 ceph-mon[36349]: pgmap v47: 97 pgs: 97 active+clean; 5.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 327 KiB/s wr, 28 op/s 2024-03-26T06:39:00.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:00 smithi170 ceph-mon[36912]: pgmap v47: 97 pgs: 97 active+clean; 5.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 327 KiB/s wr, 28 op/s 2024-03-26T06:39:00.693 INFO:tasks.workunit.client.0.smithi038.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-26T06:39:01.699 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether build environment is sane... yes 2024-03-26T06:39:01.704 INFO:tasks.workunit.client.0.smithi038.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-26T06:39:01.704 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gawk... gawk 2024-03-26T06:39:01.719 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether make sets $(MAKE)... yes 2024-03-26T06:39:01.751 INFO:tasks.workunit.client.0.smithi038.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-26T06:39:01.751 INFO:tasks.workunit.client.0.smithi038.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-26T06:39:01.762 INFO:tasks.workunit.client.0.smithi038.stdout:checking for style of include used by make... GNU 2024-03-26T06:39:01.767 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc... gcc 2024-03-26T06:39:01.848 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether the C compiler works... yes 2024-03-26T06:39:01.848 INFO:tasks.workunit.client.0.smithi038.stdout:checking for C compiler default output file name... a.out 2024-03-26T06:39:01.890 INFO:tasks.workunit.client.0.smithi038.stdout:checking for suffix of executables... 2024-03-26T06:39:01.928 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are cross compiling... no 2024-03-26T06:39:01.967 INFO:tasks.workunit.client.0.smithi038.stdout:checking for suffix of object files... o 2024-03-26T06:39:01.990 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are using the GNU C compiler... yes 2024-03-26T06:39:02.010 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether gcc accepts -g... yes 2024-03-26T06:39:02.048 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-26T06:39:02.144 INFO:tasks.workunit.client.0.smithi038.stdout:checking dependency style of gcc... gcc3 2024-03-26T06:39:02.187 INFO:tasks.workunit.client.0.smithi038.stdout:checking how to run the C preprocessor... gcc -E 2024-03-26T06:39:02.231 INFO:tasks.workunit.client.0.smithi038.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-26T06:39:02.233 INFO:tasks.workunit.client.0.smithi038.stdout:checking for egrep... /usr/bin/grep -E 2024-03-26T06:39:02.356 INFO:tasks.workunit.client.0.smithi038.stdout:checking for ANSI C header files... yes 2024-03-26T06:39:02.389 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/types.h... yes 2024-03-26T06:39:02.426 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/stat.h... yes 2024-03-26T06:39:02.464 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdlib.h... yes 2024-03-26T06:39:02.502 INFO:tasks.workunit.client.0.smithi038.stdout:checking for string.h... yes 2024-03-26T06:39:02.544 INFO:tasks.workunit.client.0.smithi038.stdout:checking for memory.h... yes 2024-03-26T06:39:02.582 INFO:tasks.workunit.client.0.smithi038.stdout:checking for strings.h... yes 2024-03-26T06:39:02.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:02 smithi038 ceph-mon[32265]: pgmap v48: 97 pgs: 97 active+clean; 5.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 329 KiB/s wr, 28 op/s 2024-03-26T06:39:02.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:02 smithi038 ceph-mon[32265]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:39:02.626 INFO:tasks.workunit.client.0.smithi038.stdout:checking for inttypes.h... yes 2024-03-26T06:39:02.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:02 smithi121 ceph-mon[36349]: pgmap v48: 97 pgs: 97 active+clean; 5.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 329 KiB/s wr, 28 op/s 2024-03-26T06:39:02.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:02 smithi121 ceph-mon[36349]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:39:02.665 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdint.h... yes 2024-03-26T06:39:02.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:02 smithi170 ceph-mon[36912]: pgmap v48: 97 pgs: 97 active+clean; 5.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 329 KiB/s wr, 28 op/s 2024-03-26T06:39:02.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:02 smithi170 ceph-mon[36912]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:39:02.706 INFO:tasks.workunit.client.0.smithi038.stdout:checking for unistd.h... yes 2024-03-26T06:39:02.752 INFO:tasks.workunit.client.0.smithi038.stdout:checking minix/config.h usability... no 2024-03-26T06:39:02.778 INFO:tasks.workunit.client.0.smithi038.stdout:checking minix/config.h presence... no 2024-03-26T06:39:02.778 INFO:tasks.workunit.client.0.smithi038.stdout:checking for minix/config.h... no 2024-03-26T06:39:02.809 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-03-26T06:39:02.809 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc... (cached) gcc 2024-03-26T06:39:02.835 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-26T06:39:02.835 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-26T06:39:02.835 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-26T06:39:02.835 INFO:tasks.workunit.client.0.smithi038.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-26T06:39:02.873 INFO:tasks.workunit.client.0.smithi038.stdout:checking for main in -lm... yes 2024-03-26T06:39:02.912 INFO:tasks.workunit.client.0.smithi038.stdout:checking for main in -lpthread... yes 2024-03-26T06:39:02.913 INFO:tasks.workunit.client.0.smithi038.stdout:checking for ANSI C header files... (cached) yes 2024-03-26T06:39:02.943 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-26T06:39:02.991 INFO:tasks.workunit.client.0.smithi038.stdout:checking pthread.h usability... yes 2024-03-26T06:39:03.009 INFO:tasks.workunit.client.0.smithi038.stdout:checking pthread.h presence... yes 2024-03-26T06:39:03.009 INFO:tasks.workunit.client.0.smithi038.stdout:checking for pthread.h... yes 2024-03-26T06:39:03.055 INFO:tasks.workunit.client.0.smithi038.stdout:checking fcntl.h usability... yes 2024-03-26T06:39:03.071 INFO:tasks.workunit.client.0.smithi038.stdout:checking fcntl.h presence... yes 2024-03-26T06:39:03.072 INFO:tasks.workunit.client.0.smithi038.stdout:checking for fcntl.h... yes 2024-03-26T06:39:03.119 INFO:tasks.workunit.client.0.smithi038.stdout:checking limits.h usability... yes 2024-03-26T06:39:03.133 INFO:tasks.workunit.client.0.smithi038.stdout:checking limits.h presence... yes 2024-03-26T06:39:03.133 INFO:tasks.workunit.client.0.smithi038.stdout:checking for limits.h... yes 2024-03-26T06:39:03.149 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stdint.h... (cached) yes 2024-03-26T06:39:03.182 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/time.h usability... yes 2024-03-26T06:39:03.195 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/time.h presence... yes 2024-03-26T06:39:03.196 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/time.h... yes 2024-03-26T06:39:03.213 INFO:tasks.workunit.client.0.smithi038.stdout:checking for unistd.h... (cached) yes 2024-03-26T06:39:03.250 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/vfs.h usability... yes 2024-03-26T06:39:03.265 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/vfs.h presence... yes 2024-03-26T06:39:03.265 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/vfs.h... yes 2024-03-26T06:39:03.313 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/limits.h usability... no 2024-03-26T06:39:03.336 INFO:tasks.workunit.client.0.smithi038.stdout:checking sys/limits.h presence... no 2024-03-26T06:39:03.336 INFO:tasks.workunit.client.0.smithi038.stdout:checking for sys/limits.h... no 2024-03-26T06:39:03.357 INFO:tasks.workunit.client.0.smithi038.stdout:checking for an ANSI C-conforming const... yes 2024-03-26T06:39:03.381 INFO:tasks.workunit.client.0.smithi038.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-26T06:39:03.446 INFO:tasks.workunit.client.0.smithi038.stdout:checking for system... yes 2024-03-26T06:39:03.506 INFO:tasks.workunit.client.0.smithi038.stdout:checking for gettimeofday... yes 2024-03-26T06:39:03.559 INFO:tasks.workunit.client.0.smithi038.stdout:checking for mkdir... yes 2024-03-26T06:39:03.611 INFO:tasks.workunit.client.0.smithi038.stdout:checking for strerror... yes 2024-03-26T06:39:03.664 INFO:tasks.workunit.client.0.smithi038.stdout:checking for open64... yes 2024-03-26T06:39:03.717 INFO:tasks.workunit.client.0.smithi038.stdout:checking for stat64... yes 2024-03-26T06:39:03.770 INFO:tasks.workunit.client.0.smithi038.stdout:checking for fseeko64... yes 2024-03-26T06:39:03.823 INFO:tasks.workunit.client.0.smithi038.stdout:checking for lrand48_r... yes 2024-03-26T06:39:03.875 INFO:tasks.workunit.client.0.smithi038.stdout:checking for srand48_r... yes 2024-03-26T06:39:03.899 INFO:tasks.workunit.client.0.smithi038.stdout:configure: creating ./config.status 2024-03-26T06:39:03.957 INFO:tasks.workunit.client.0.smithi038.stdout: /bin/sh ./config.status 2024-03-26T06:39:04.001 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: creating Makefile 2024-03-26T06:39:04.012 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: creating config.h 2024-03-26T06:39:04.034 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: executing depfiles commands 2024-03-26T06:39:04.044 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: executing default commands 2024-03-26T06:39:04.072 INFO:tasks.workunit.client.0.smithi038.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-03-26T06:39:04.223 INFO:tasks.workunit.client.0.smithi038.stdout:rm -f stamp-h1 2024-03-26T06:39:04.226 INFO:tasks.workunit.client.0.smithi038.stdout:touch config.h.in 2024-03-26T06:39:04.227 INFO:tasks.workunit.client.0.smithi038.stdout:cd . && /bin/sh ./config.status config.h 2024-03-26T06:39:04.265 INFO:tasks.workunit.client.0.smithi038.stdout:config.status: creating config.h 2024-03-26T06:39:04.288 INFO:tasks.workunit.client.0.smithi038.stdout:make all-am 2024-03-26T06:39:04.294 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-26T06:39:04.294 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-03-26T06:39:04.462 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-03-26T06:39:04.465 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-03-26T06:39:04.552 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-03-26T06:39:04.555 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-03-26T06:39:04.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:04 smithi038 ceph-mon[32265]: pgmap v49: 97 pgs: 97 active+clean; 5.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 329 KiB/s wr, 26 op/s 2024-03-26T06:39:04.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:04 smithi038 ceph-mon[32265]: mgrmap e26: x(active, since 90s), standbys: z, y 2024-03-26T06:39:04.634 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-03-26T06:39:04.637 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-03-26T06:39:04.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:04 smithi121 ceph-mon[36349]: pgmap v49: 97 pgs: 97 active+clean; 5.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 329 KiB/s wr, 26 op/s 2024-03-26T06:39:04.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:04 smithi121 ceph-mon[36349]: mgrmap e26: x(active, since 90s), standbys: z, y 2024-03-26T06:39:04.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:04 smithi170 ceph-mon[36912]: pgmap v49: 97 pgs: 97 active+clean; 5.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 329 KiB/s wr, 26 op/s 2024-03-26T06:39:04.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:04 smithi170 ceph-mon[36912]: mgrmap e26: x(active, since 90s), standbys: z, y 2024-03-26T06:39:04.723 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-03-26T06:39:04.727 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-03-26T06:39:04.847 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-03-26T06:39:04.851 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-03-26T06:39:04.896 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-03-26T06:39:04.899 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-03-26T06:39:04.938 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-03-26T06:39:04.941 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-03-26T06:39:04.984 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-03-26T06:39:04.989 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-03-26T06:39:05.191 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-03-26T06:39:05.195 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-03-26T06:39:05.324 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-03-26T06:39:05.328 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-03-26T06:39:05.469 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-03-26T06:39:05.473 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-03-26T06:39:05.539 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-03-26T06:39:05.543 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-03-26T06:39:05.609 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-03-26T06:39:05.613 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-03-26T06:39:05.712 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-03-26T06:39:05.716 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-03-26T06:39:05.744 INFO:tasks.workunit.client.0.smithi038.stderr:In file included from parser.h:22, 2024-03-26T06:39:05.744 INFO:tasks.workunit.client.0.smithi038.stderr: from parser.c:25: 2024-03-26T06:39:05.744 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c: In function ‘init_filesys’: 2024-03-26T06:39:05.744 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-26T06:39:05.745 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-26T06:39:05.746 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-03-26T06:39:05.747 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-03-26T06:39:05.748 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-26T06:39:05.748 INFO:tasks.workunit.client.0.smithi038.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-26T06:39:05.748 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-03-26T06:39:05.748 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-26T06:39:05.748 INFO:tasks.workunit.client.0.smithi038.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-26T06:39:05.748 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-26T06:39:05.748 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-26T06:39:05.748 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-26T06:39:05.748 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-03-26T06:39:05.748 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-26T06:39:05.748 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c: In function ‘init_tg_stats’: 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-26T06:39:05.749 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-03-26T06:39:05.750 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-26T06:39:05.750 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-26T06:39:05.750 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-03-26T06:39:05.750 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-26T06:39:05.750 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-03-26T06:39:05.750 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-03-26T06:39:05.750 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-26T06:39:05.750 INFO:tasks.workunit.client.0.smithi038.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-26T06:39:05.750 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-03-26T06:39:05.750 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-26T06:39:05.750 INFO:tasks.workunit.client.0.smithi038.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-03-26T06:39:05.751 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-26T06:39:05.752 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr: | ^ 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~ 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr: 36 | container_of(ptr, type, member) 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~ 2024-03-26T06:39:05.753 INFO:tasks.workunit.client.0.smithi038.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-26T06:39:05.754 INFO:tasks.workunit.client.0.smithi038.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-26T06:39:05.754 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~ 2024-03-26T06:39:05.754 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-26T06:39:05.754 INFO:tasks.workunit.client.0.smithi038.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-26T06:39:05.754 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-26T06:39:05.792 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c: In function ‘get_optstr’: 2024-03-26T06:39:05.792 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-03-26T06:39:05.792 INFO:tasks.workunit.client.0.smithi038.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-03-26T06:39:05.792 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-26T06:39:05.792 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:200:14: note: source object declared here 2024-03-26T06:39:05.793 INFO:tasks.workunit.client.0.smithi038.stderr: 200 | char temp[BUFSIZE]; 2024-03-26T06:39:05.793 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~ 2024-03-26T06:39:05.798 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c: In function ‘size64_convert’: 2024-03-26T06:39:05.798 INFO:tasks.workunit.client.0.smithi038.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-03-26T06:39:05.798 INFO:tasks.workunit.client.0.smithi038.stderr: 119 | memcpy(unit, "\0", 3); 2024-03-26T06:39:05.799 INFO:tasks.workunit.client.0.smithi038.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-03-26T06:39:06.088 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-03-26T06:39:06.093 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-03-26T06:39:06.147 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-03-26T06:39:06.151 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-03-26T06:39:06.237 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-03-26T06:39:06.241 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-03-26T06:39:06.269 INFO:tasks.workunit.client.0.smithi038.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-03-26T06:39:06.274 INFO:tasks.workunit.client.0.smithi038.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-03-26T06:39:06.307 INFO:tasks.workunit.client.0.smithi038.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-26T06:39:06.307 INFO:tasks.workunit.client.0.smithi038.stderr:+ cd .. 2024-03-26T06:39:06.307 INFO:tasks.workunit.client.0.smithi038.stderr:+ mkdir tmp 2024-03-26T06:39:06.309 INFO:tasks.workunit.client.0.smithi038.stderr:+ cd tmp 2024-03-26T06:39:06.309 INFO:tasks.workunit.client.0.smithi038.stderr:+ for f in $mydir/*.ffsb 2024-03-26T06:39:06.309 INFO:tasks.workunit.client.0.smithi038.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-03-26T06:39:06.310 INFO:tasks.workunit.client.0.smithi038.stdout:FFSB version 6.0-RC2 started 2024-03-26T06:39:06.310 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-03-26T06:39:06.313 INFO:tasks.workunit.client.0.smithi038.stdout:benchmark time = 300 2024-03-26T06:39:06.313 INFO:tasks.workunit.client.0.smithi038.stdout:ThreadGroup 0 2024-03-26T06:39:06.313 INFO:tasks.workunit.client.0.smithi038.stdout:================ 2024-03-26T06:39:06.313 INFO:tasks.workunit.client.0.smithi038.stdout: num_threads = 32 2024-03-26T06:39:06.313 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-03-26T06:39:06.313 INFO:tasks.workunit.client.0.smithi038.stdout: read_random = off 2024-03-26T06:39:06.313 INFO:tasks.workunit.client.0.smithi038.stdout: read_size = 0 (0B) 2024-03-26T06:39:06.313 INFO:tasks.workunit.client.0.smithi038.stdout: read_blocksize = 0 (0B) 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: read_skip = off 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: read_skipsize = 0 (0B) 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: write_random = on 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: write_size = 5242880 (5MB) 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: fsync_file = 0 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: write_blocksize = 4096 (4KB) 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: wait time = 0 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: op weights 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: read = 0 (0.00%) 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: readall = 0 (0.00%) 2024-03-26T06:39:06.314 INFO:tasks.workunit.client.0.smithi038.stdout: write = 1 (100.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: create = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: append = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: delete = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: metaop = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: createdir = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: stat = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: writeall = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: writeall_fsync = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: open_close = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: write_fsync = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: create_fsync = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: append_fsync = 0 (0.00%) 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-03-26T06:39:06.315 INFO:tasks.workunit.client.0.smithi038.stdout:FileSystem . 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout:========== 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: num_dirs = 0 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: starting files = 128 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: min file size = 104857600 (100MB) 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: max file size = 104857600 (100MB) 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: directio = off 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: alignedio = on 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: bufferedio = off 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: aging is off 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: current utilization = 0.00% 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-03-26T06:39:06.316 INFO:tasks.workunit.client.0.smithi038.stdout:checking existing fs: . 2024-03-26T06:39:06.317 INFO:tasks.workunit.client.0.smithi038.stdout:recreating new fileset 2024-03-26T06:39:06.317 INFO:tasks.workunit.client.0.smithi038.stderr:opendir: No such file or directory 2024-03-26T06:39:06.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:06 smithi038 ceph-mon[32265]: pgmap v50: 97 pgs: 97 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 602 KiB/s wr, 55 op/s 2024-03-26T06:39:06.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:06 smithi121 ceph-mon[36349]: pgmap v50: 97 pgs: 97 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 602 KiB/s wr, 55 op/s 2024-03-26T06:39:06.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:06 smithi170 ceph-mon[36912]: pgmap v50: 97 pgs: 97 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 602 KiB/s wr, 55 op/s 2024-03-26T06:39:07.558 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.555+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/3556077483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc069310 msgr2=0x7fd6bc10e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:07.558 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.555+0000 7fd6c3fcd640 1 --2- 172.21.15.38:0/3556077483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc069310 0x7fd6bc10e180 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4009ca0 tx=0x7fd6b402c180 comp rx=0 tx=0).stop 2024-03-26T06:39:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.555+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/3556077483 shutdown_connections 2024-03-26T06:39:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.555+0000 7fd6c3fcd640 1 --2- 172.21.15.38:0/3556077483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc069310 0x7fd6bc10e180 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.555+0000 7fd6c3fcd640 1 --2- 172.21.15.38:0/3556077483 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6bc068970 0x7fd6bc068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.555+0000 7fd6c3fcd640 1 --2- 172.21.15.38:0/3556077483 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6bc1073e0 0x7fd6bc1077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.555+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/3556077483 >> 172.21.15.38:0/3556077483 conn(0x7fd6bc077040 msgr2=0x7fd6bc077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.556+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/3556077483 shutdown_connections 2024-03-26T06:39:07.559 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.556+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/3556077483 wait complete. 2024-03-26T06:39:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.556+0000 7fd6c3fcd640 1 Processor -- start 2024-03-26T06:39:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.556+0000 7fd6c3fcd640 1 -- start start 2024-03-26T06:39:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c3fcd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6bc068970 0x7fd6bc19d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.561 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c3fcd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc069310 0x7fd6bc19dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c3fcd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6bc1073e0 0x7fd6bc1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c3fcd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc111670 con 0x7fd6bc069310 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c3fcd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1114f0 con 0x7fd6bc068970 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c3fcd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1117f0 con 0x7fd6bc1073e0 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c1568640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc069310 0x7fd6bc19dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c1568640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc069310 0x7fd6bc19dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59854/0 (socket says 172.21.15.38:59854) 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c1568640 1 -- 172.21.15.38:0/2727940978 learned_addr learned my addr 172.21.15.38:0/2727940978 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c1d69640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6bc068970 0x7fd6bc19d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c256a640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6bc1073e0 0x7fd6bc1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c1568640 1 -- 172.21.15.38:0/2727940978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6bc1073e0 msgr2=0x7fd6bc1979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c1568640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6bc1073e0 0x7fd6bc1979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c1568640 1 -- 172.21.15.38:0/2727940978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6bc068970 msgr2=0x7fd6bc19d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:07.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c1568640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6bc068970 0x7fd6bc19d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c1568640 1 -- 172.21.15.38:0/2727940978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc198130 con 0x7fd6bc069310 2024-03-26T06:39:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c1d69640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6bc068970 0x7fd6bc19d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:39:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.557+0000 7fd6c1568640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc069310 0x7fd6bc19dcf0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00d770 tx=0x7fd6ac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.558+0000 7fd6b2ffd640 1 -- 172.21.15.38:0/2727940978 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac017070 con 0x7fd6bc069310 2024-03-26T06:39:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.558+0000 7fd6b2ffd640 1 -- 172.21.15.38:0/2727940978 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6ac005500 con 0x7fd6bc069310 2024-03-26T06:39:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.558+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/2727940978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6bc198420 con 0x7fd6bc069310 2024-03-26T06:39:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.558+0000 7fd6b2ffd640 1 -- 172.21.15.38:0/2727940978 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac004e00 con 0x7fd6bc069310 2024-03-26T06:39:07.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.558+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/2727940978 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6bc1a4670 con 0x7fd6bc069310 2024-03-26T06:39:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.560+0000 7fd6b2ffd640 1 -- 172.21.15.38:0/2727940978 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd6ac010460 con 0x7fd6bc069310 2024-03-26T06:39:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.560+0000 7fd6c6baa740 1 Processor -- start 2024-03-26T06:39:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.560+0000 7fd6c6baa740 1 -- start start 2024-03-26T06:39:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.560+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 0x560c91c1aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.560+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1afd0 0x560c91c1b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.560+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c24550 0x560c91c26940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.560+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c91b7f5f0 con 0x560c91c1bd40 2024-03-26T06:39:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.560+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c91b28930 con 0x560c91c1afd0 2024-03-26T06:39:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.560+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c91b7b690 con 0x560c91c24550 2024-03-26T06:39:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6b2ffd640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd6a80a2a10 0x7fd6a80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c1d69640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1afd0 0x560c91c1b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c256a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 0x560c91c1aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c1d69640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1afd0 0x560c91c1b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:45612/0 (socket says 172.21.15.38:45612) 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c1d69640 1 -- 172.21.15.38:0/2997760528 learned_addr learned my addr 172.21.15.38:0/2997760528 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c256a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 0x560c91c1aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59870/0 (socket says 172.21.15.38:59870) 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c256a640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd6a80a2a10 0x7fd6a80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c1568640 1 --2- 172.21.15.38:0/2997760528 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c24550 0x560c91c26940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6b2ffd640 1 -- 172.21.15.38:0/2727940978 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fd6ac0ce950 con 0x7fd6bc069310 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c1d69640 1 -- 172.21.15.38:0/2997760528 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c24550 msgr2=0x560c91c26940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c1d69640 1 --2- 172.21.15.38:0/2997760528 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c24550 0x560c91c26940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c1d69640 1 -- 172.21.15.38:0/2997760528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 msgr2=0x560c91c1aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c1d69640 1 --2- 172.21.15.38:0/2997760528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 0x560c91c1aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c1d69640 1 -- 172.21.15.38:0/2997760528 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c91894210 con 0x560c91c1afd0 2024-03-26T06:39:07.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c1d69640 1 --2- 172.21.15.38:0/2997760528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1afd0 0x560c91c1b450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8010400 tx=0x7fd6b80108c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98a3b525e5073719 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6c256a640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd6a80a2a10 0x7fd6a80a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4006010 tx=0x7fd6b40095c0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6a7fff640 1 -- 172.21.15.38:0/2997760528 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8004220 con 0x560c91c1afd0 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6a7fff640 1 -- 172.21.15.38:0/2997760528 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6b80043c0 con 0x560c91c1afd0 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.561+0000 7fd6a7fff640 1 -- 172.21.15.38:0/2997760528 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8003850 con 0x560c91c1afd0 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.562+0000 7fd6c6baa740 1 -- 172.21.15.38:0/2997760528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1afd0 msgr2=0x560c91c1b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.562+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/2997760528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1afd0 0x560c91c1b450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8010400 tx=0x7fd6b80108c0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.562+0000 7fd6c6baa740 1 -- 172.21.15.38:0/2997760528 shutdown_connections 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.562+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/2997760528 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c24550 0x560c91c26940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.562+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/2997760528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1afd0 0x560c91c1b450 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.562+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/2997760528 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 0x560c91c1aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.562+0000 7fd6c6baa740 1 -- 172.21.15.38:0/2997760528 >> 172.21.15.38:0/2997760528 conn(0x560c91be25b0 msgr2=0x560c91c15cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.562+0000 7fd6c6baa740 1 -- 172.21.15.38:0/2997760528 shutdown_connections 2024-03-26T06:39:07.572 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.562+0000 7fd6c6baa740 1 -- 172.21.15.38:0/2997760528 wait complete. 2024-03-26T06:39:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c6baa740 1 -- start start 2024-03-26T06:39:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c6baa740 1 Processor -- start 2024-03-26T06:39:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1afd0 0x560c91c21590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1bd40 0x560c91c21ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c24550 0x560c91c22010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c91c27280 con 0x560c91c24550 2024-03-26T06:39:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c91b3bd90 con 0x560c91c1bd40 2024-03-26T06:39:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c91c27400 con 0x560c91c1afd0 2024-03-26T06:39:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c6baa740 10 client.0 _ll_register_callbacks cb 0x560c91893a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:39:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c256a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c24550 0x560c91c22010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c1568640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1bd40 0x560c91c21ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c256a640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c24550 0x560c91c22010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59876/0 (socket says 172.21.15.38:59876) 2024-03-26T06:39:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c256a640 1 -- 172.21.15.38:0/764884209 learned_addr learned my addr 172.21.15.38:0/764884209 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.563+0000 7fd6c1d69640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1afd0 0x560c91c21590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c1568640 1 -- 172.21.15.38:0/764884209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1afd0 msgr2=0x560c91c21590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c1568640 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1afd0 0x560c91c21590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c1568640 1 -- 172.21.15.38:0/764884209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c24550 msgr2=0x560c91c22010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c1568640 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c24550 0x560c91c22010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c1568640 1 -- 172.21.15.38:0/764884209 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c91960620 con 0x560c91c1bd40 2024-03-26T06:39:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c256a640 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c24550 0x560c91c22010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:39:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c1568640 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1bd40 0x560c91c21ad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0cc090 tx=0x7fd6ac0077b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c5f8a4b1762fb15 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6a57fa640 1 -- 172.21.15.38:0/764884209 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac09a900 con 0x560c91c1bd40 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6a57fa640 1 -- 172.21.15.38:0/764884209 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fd6ac097610 con 0x560c91c1bd40 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c6baa740 1 -- 172.21.15.38:0/764884209 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6b80043c0 con 0x560c91c1bd40 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6a57fa640 1 -- 172.21.15.38:0/764884209 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6ac0046c0 con 0x560c91c1bd40 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c6baa740 10 client.25148 fetch_fsmap learned FSMap version 27 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c6baa740 1 -- 172.21.15.38:0/764884209 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x560c91c335d0 con 0x560c91c1bd40 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6a57fa640 1 -- 172.21.15.38:0/764884209 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6b80043c0 con 0x560c91c1bd40 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6a57fa640 1 -- 172.21.15.38:0/764884209 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd6ac09a3f0 con 0x560c91c1bd40 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c6baa740 10 client.25148 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c6baa740 10 client.25148 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c6baa740 20 client.25148 populate_metadata read hostname 'smithi038' 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.564+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560c91c34a90 0x560c91c36e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.565+0000 7fd6c6baa740 4 client.25148 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:39:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.565+0000 7fd6c6baa740 1 -- 172.21.15.38:0/764884209 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c91b8cb30 con 0x560c91c34a90 2024-03-26T06:39:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.565+0000 7fd6c1d69640 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560c91c34a90 0x560c91c36e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.565+0000 7fd6c1d69640 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560c91c34a90 0x560c91c36e80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=528fdb9c7af7ec06 server_cookie=edf321e832a54b46 in_seq=0 out_seq=0 2024-03-26T06:39:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.565+0000 7fd6a57fa640 10 client.25148 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6a57fa640 1 -- 172.21.15.38:0/764884209 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560c91b8cb30 con 0x560c91c34a90 2024-03-26T06:39:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6a57fa640 10 client.25148 handle_command_reply: tid=0 2024-03-26T06:39:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 1 client.25148 shutdown 2024-03-26T06:39:07.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 1 -- 172.21.15.38:0/764884209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1bd40 msgr2=0x560c91c21ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1bd40 0x560c91c21ad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac0cc090 tx=0x7fd6ac0077b0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 1 -- 172.21.15.38:0/764884209 shutdown_connections 2024-03-26T06:39:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560c91c34a90 0x560c91c36e80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c24550 0x560c91c22010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c1bd40 0x560c91c21ad0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/764884209 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1afd0 0x560c91c21590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 1 -- 172.21.15.38:0/764884209 >> 172.21.15.38:0/764884209 conn(0x560c91be25b0 msgr2=0x560c91c17a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6a57fa640 0 client.25148 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 1 -- 172.21.15.38:0/764884209 shutdown_connections 2024-03-26T06:39:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 1 -- 172.21.15.38:0/764884209 wait complete. 2024-03-26T06:39:07.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.566+0000 7fd6c6baa740 20 client.25148 trim_cache size 0 max 16384 2024-03-26T06:39:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c6baa740 1 Processor -- start 2024-03-26T06:39:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c6baa740 1 -- start start 2024-03-26T06:39:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1afd0 0x560c91c23cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 0x560c91c2b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c24550 0x560c91c2bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1af230 con 0x560c91c1bd40 2024-03-26T06:39:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1acd70 con 0x560c91c24550 2024-03-26T06:39:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1aae10 con 0x560c91c1afd0 2024-03-26T06:39:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c1568640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 0x560c91c2b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c1568640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 0x560c91c2b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59888/0 (socket says 172.21.15.38:59888) 2024-03-26T06:39:07.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c1568640 1 -- 172.21.15.38:0/2240143653 learned_addr learned my addr 172.21.15.38:0/2240143653 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c1d69640 1 --2- 172.21.15.38:0/2240143653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1afd0 0x560c91c23cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c256a640 1 --2- 172.21.15.38:0/2240143653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c24550 0x560c91c2bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c1568640 1 -- 172.21.15.38:0/2240143653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1afd0 msgr2=0x560c91c23cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c1568640 1 --2- 172.21.15.38:0/2240143653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1afd0 0x560c91c23cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c1568640 1 -- 172.21.15.38:0/2240143653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c24550 msgr2=0x560c91c2bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c1568640 1 --2- 172.21.15.38:0/2240143653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c24550 0x560c91c2bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c1568640 1 -- 172.21.15.38:0/2240143653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c91c3c0b0 con 0x560c91c1bd40 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c256a640 1 --2- 172.21.15.38:0/2240143653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c24550 0x560c91c2bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.580+0000 7fd6c1568640 1 --2- 172.21.15.38:0/2240143653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 0x560c91c2b6a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac097400 tx=0x7fd6ac009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3e6d970e2031783 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6a77fe640 1 -- 172.21.15.38:0/2240143653 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac020070 con 0x560c91c1bd40 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6a77fe640 1 -- 172.21.15.38:0/2240143653 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6ac09b6a0 con 0x560c91c1bd40 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6a77fe640 1 -- 172.21.15.38:0/2240143653 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ac096cf0 con 0x560c91c1bd40 2024-03-26T06:39:07.589 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6c6baa740 1 -- 172.21.15.38:0/2240143653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 msgr2=0x560c91c2b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/2240143653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 0x560c91c2b6a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac097400 tx=0x7fd6ac009c30 comp rx=0 tx=0).stop 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6c6baa740 1 -- 172.21.15.38:0/2240143653 shutdown_connections 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/2240143653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c24550 0x560c91c2bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/2240143653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1bd40 0x560c91c2b6a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/2240143653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1afd0 0x560c91c23cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6c6baa740 1 -- 172.21.15.38:0/2240143653 >> 172.21.15.38:0/2240143653 conn(0x560c91be25b0 msgr2=0x560c91c1c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6c6baa740 1 -- 172.21.15.38:0/2240143653 shutdown_connections 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.581+0000 7fd6c6baa740 1 -- 172.21.15.38:0/2240143653 wait complete. 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c6baa740 1 -- start start 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c6baa740 1 Processor -- start 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1afd0 0x560c91c2b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1bd40 0x560c91c27a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c6baa740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c24550 0x560c91c27fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1ab410 con 0x560c91c1afd0 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1ac710 con 0x560c91c24550 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c6baa740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6bc1aba50 con 0x560c91c1bd40 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c6baa740 10 client.0 _ll_register_callbacks cb 0x7fd6b800ba60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c1d69640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1afd0 0x560c91c2b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c1d69640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1afd0 0x560c91c2b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59902/0 (socket says 172.21.15.38:59902) 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c1d69640 1 -- 172.21.15.38:0/331983330 learned_addr learned my addr 172.21.15.38:0/331983330 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c1568640 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1bd40 0x560c91c27a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.582+0000 7fd6c256a640 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c24550 0x560c91c27fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6c1d69640 1 -- 172.21.15.38:0/331983330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1bd40 msgr2=0x560c91c27a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6c1d69640 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1bd40 0x560c91c27a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6c1d69640 1 -- 172.21.15.38:0/331983330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c24550 msgr2=0x560c91c27fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6c1d69640 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c24550 0x560c91c27fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.591 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6c1d69640 1 -- 172.21.15.38:0/331983330 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c91c284f0 con 0x560c91c1afd0 2024-03-26T06:39:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6c1d69640 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1afd0 0x560c91c2b6f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8010dc0 tx=0x7fd6b800bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b381971bb3cbc76a server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:39:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6a57fa640 1 -- 172.21.15.38:0/331983330 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b802d070 con 0x560c91c1afd0 2024-03-26T06:39:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6c6baa740 1 -- 172.21.15.38:0/331983330 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6ac09b6a0 con 0x560c91c1afd0 2024-03-26T06:39:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6a57fa640 1 -- 172.21.15.38:0/331983330 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fd6b8027870 con 0x560c91c1afd0 2024-03-26T06:39:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6a57fa640 1 -- 172.21.15.38:0/331983330 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6b8038050 con 0x560c91c1afd0 2024-03-26T06:39:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6c6baa740 10 client.15522 fetch_fsmap learned FSMap version 27 2024-03-26T06:39:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6c6baa740 1 -- 172.21.15.38:0/331983330 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x560c91c8bc90 con 0x560c91c1afd0 2024-03-26T06:39:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.583+0000 7fd6a57fa640 1 -- 172.21.15.38:0/331983330 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6ac09b6a0 con 0x560c91c1afd0 2024-03-26T06:39:07.592 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.584+0000 7fd6a57fa640 1 -- 172.21.15.38:0/331983330 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fd6b8011560 con 0x560c91c1afd0 2024-03-26T06:39:07.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.584+0000 7fd6c6baa740 10 client.15522 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:39:07.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.584+0000 7fd6c6baa740 10 client.15522 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:39:07.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.584+0000 7fd6c6baa740 20 client.15522 populate_metadata read hostname 'smithi038' 2024-03-26T06:39:07.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.584+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560c91c8c0b0 0x560c91c8e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:07.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.584+0000 7fd6c6baa740 4 client.15522 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:39:07.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.584+0000 7fd6c6baa740 1 -- 172.21.15.38:0/331983330 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560c91b86e80 con 0x560c91c8c0b0 2024-03-26T06:39:07.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.584+0000 7fd6c1568640 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560c91c8c0b0 0x560c91c8e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:07.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.584+0000 7fd6c1568640 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560c91c8c0b0 0x560c91c8e4d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10b238c40691076f server_cookie=d275a0ab073b5acc in_seq=0 out_seq=0 2024-03-26T06:39:07.593 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.584+0000 7fd6a57fa640 10 client.15522 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.675+0000 7fd6a57fa640 1 -- 172.21.15.38:0/331983330 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560c91b86e80 con 0x560c91c8c0b0 2024-03-26T06:39:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.675+0000 7fd6a57fa640 10 client.15522 handle_command_reply: tid=0 2024-03-26T06:39:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.675+0000 7fd6c6baa740 1 client.15522 shutdown 2024-03-26T06:39:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c6baa740 1 -- 172.21.15.38:0/331983330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1afd0 msgr2=0x560c91c2b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1afd0 0x560c91c2b6f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd6b8010dc0 tx=0x7fd6b800bc50 comp rx=0 tx=0).stop 2024-03-26T06:39:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c6baa740 1 -- 172.21.15.38:0/331983330 shutdown_connections 2024-03-26T06:39:07.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560c91c8c0b0 0x560c91c8e4d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560c91c24550 0x560c91c27fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560c91c1bd40 0x560c91c27a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c6baa740 1 --2- 172.21.15.38:0/331983330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560c91c1afd0 0x560c91c2b6f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6a57fa640 0 client.15522 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c6baa740 1 -- 172.21.15.38:0/331983330 >> 172.21.15.38:0/331983330 conn(0x560c91be25b0 msgr2=0x560c91c1a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c6baa740 1 -- 172.21.15.38:0/331983330 shutdown_connections 2024-03-26T06:39:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c6baa740 1 -- 172.21.15.38:0/331983330 wait complete. 2024-03-26T06:39:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c6baa740 20 client.15522 trim_cache size 0 max 16384 2024-03-26T06:39:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:39:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:39:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:39:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:39:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/2727940978 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd6a80a2a10 msgr2=0x7fd6a80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c3fcd640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd6a80a2a10 0x7fd6a80a4ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd6b4006010 tx=0x7fd6b40095c0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/2727940978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc069310 msgr2=0x7fd6bc19dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c3fcd640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc069310 0x7fd6bc19dcf0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fd6ac00d770 tx=0x7fd6ac00dc30 comp rx=0 tx=0).stop 2024-03-26T06:39:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/2727940978 shutdown_connections 2024-03-26T06:39:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c3fcd640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd6a80a2a10 0x7fd6a80a4ed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c3fcd640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd6bc1073e0 0x7fd6bc1979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c3fcd640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd6bc069310 0x7fd6bc19dcf0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c3fcd640 1 --2- 172.21.15.38:0/2727940978 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6bc068970 0x7fd6bc19d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.676+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/2727940978 >> 172.21.15.38:0/2727940978 conn(0x7fd6bc077040 msgr2=0x7fd6bc0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.677+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/2727940978 shutdown_connections 2024-03-26T06:39:07.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:07.677+0000 7fd6c3fcd640 1 -- 172.21.15.38:0/2727940978 wait complete. 2024-03-26T06:39:08.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:08 smithi038 ceph-mon[32265]: pgmap v51: 97 pgs: 97 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 602 KiB/s wr, 52 op/s 2024-03-26T06:39:08.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:08 smithi121 ceph-mon[36349]: pgmap v51: 97 pgs: 97 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 602 KiB/s wr, 52 op/s 2024-03-26T06:39:08.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:08 smithi170 ceph-mon[36912]: pgmap v51: 97 pgs: 97 active+clean; 7.9 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 602 KiB/s wr, 52 op/s 2024-03-26T06:39:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:10 smithi038 ceph-mon[32265]: pgmap v52: 97 pgs: 97 active+clean; 338 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 29 MiB/s wr, 222 op/s 2024-03-26T06:39:10.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:10 smithi121 ceph-mon[36349]: pgmap v52: 97 pgs: 97 active+clean; 338 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 29 MiB/s wr, 222 op/s 2024-03-26T06:39:10.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:10 smithi170 ceph-mon[36912]: pgmap v52: 97 pgs: 97 active+clean; 338 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 29 MiB/s wr, 222 op/s 2024-03-26T06:39:12.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:12 smithi038 ceph-mon[32265]: pgmap v53: 97 pgs: 97 active+clean; 338 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 28 MiB/s wr, 198 op/s 2024-03-26T06:39:12.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:12 smithi121 ceph-mon[36349]: pgmap v53: 97 pgs: 97 active+clean; 338 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 28 MiB/s wr, 198 op/s 2024-03-26T06:39:12.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:12 smithi170 ceph-mon[36912]: pgmap v53: 97 pgs: 97 active+clean; 338 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 28 MiB/s wr, 198 op/s 2024-03-26T06:39:14.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:14 smithi038 ceph-mon[32265]: pgmap v54: 97 pgs: 97 active+clean; 338 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 28 MiB/s wr, 197 op/s 2024-03-26T06:39:14.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:14 smithi121 ceph-mon[36349]: pgmap v54: 97 pgs: 97 active+clean; 338 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 28 MiB/s wr, 197 op/s 2024-03-26T06:39:14.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:14 smithi170 ceph-mon[36912]: pgmap v54: 97 pgs: 97 active+clean; 338 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 28 MiB/s wr, 197 op/s 2024-03-26T06:39:16.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:16 smithi121 ceph-mon[36349]: pgmap v55: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 320 MiB/s wr, 331 op/s 2024-03-26T06:39:16.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:16 smithi170 ceph-mon[36912]: pgmap v55: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 320 MiB/s wr, 331 op/s 2024-03-26T06:39:16.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:16 smithi038 ceph-mon[32265]: pgmap v55: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 320 MiB/s wr, 331 op/s 2024-03-26T06:39:18.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:18 smithi170 ceph-mon[36912]: pgmap v56: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 320 MiB/s wr, 303 op/s 2024-03-26T06:39:18.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:18 smithi038 ceph-mon[32265]: pgmap v56: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 320 MiB/s wr, 303 op/s 2024-03-26T06:39:18.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:18 smithi121 ceph-mon[36349]: pgmap v56: 97 pgs: 97 active+clean; 3.8 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 320 MiB/s wr, 303 op/s 2024-03-26T06:39:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.799+0000 7f20b1132640 1 -- 172.21.15.38:0/2194887765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20ac1024d0 msgr2=0x7f20ac1028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.799+0000 7f20b1132640 1 --2- 172.21.15.38:0/2194887765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20ac1024d0 0x7f20ac1028b0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f209c009900 tx=0x7f209c02bc50 comp rx=0 tx=0).stop 2024-03-26T06:39:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.799+0000 7f20b1132640 1 -- 172.21.15.38:0/2194887765 shutdown_connections 2024-03-26T06:39:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.799+0000 7f20b1132640 1 --2- 172.21.15.38:0/2194887765 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac10b940 0x7f20ac10ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.799+0000 7f20b1132640 1 --2- 172.21.15.38:0/2194887765 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20ac102df0 0x7f20ac10b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.799+0000 7f20b1132640 1 --2- 172.21.15.38:0/2194887765 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20ac1024d0 0x7f20ac1028b0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.799+0000 7f20b1132640 1 -- 172.21.15.38:0/2194887765 >> 172.21.15.38:0/2194887765 conn(0x7f20ac077320 msgr2=0x7f20ac077730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:20.803 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.799+0000 7f20b1132640 1 -- 172.21.15.38:0/2194887765 shutdown_connections 2024-03-26T06:39:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.799+0000 7f20b1132640 1 -- 172.21.15.38:0/2194887765 wait complete. 2024-03-26T06:39:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.800+0000 7f20b1132640 1 Processor -- start 2024-03-26T06:39:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.800+0000 7f20b1132640 1 -- start start 2024-03-26T06:39:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.800+0000 7f20b1132640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20ac1024d0 0x7f20ac19d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.800+0000 7f20b1132640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac102df0 0x7f20ac19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.800+0000 7f20b1132640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20ac10b940 0x7f20ac197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.800+0000 7f20b1132640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac1117f0 con 0x7f20ac10b940 2024-03-26T06:39:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.800+0000 7f20b1132640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac111670 con 0x7f20ac102df0 2024-03-26T06:39:20.804 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.800+0000 7f20b1132640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac111970 con 0x7f20ac1024d0 2024-03-26T06:39:20.805 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.800+0000 7f20ab59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20ac10b940 0x7f20ac197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:20.805 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.800+0000 7f20ab59e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20ac10b940 0x7f20ac197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46684/0 (socket says 172.21.15.38:46684) 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.800+0000 7f20ab59e640 1 -- 172.21.15.38:0/3410086811 learned_addr learned my addr 172.21.15.38:0/3410086811 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.801+0000 7f20ab59e640 1 -- 172.21.15.38:0/3410086811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20ac1024d0 msgr2=0x7f20ac19d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.801+0000 7f20ab59e640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20ac1024d0 0x7f20ac19d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.801+0000 7f20ab59e640 1 -- 172.21.15.38:0/3410086811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac102df0 msgr2=0x7f20ac19ddd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.801+0000 7f20ab59e640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac102df0 0x7f20ac19ddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.801+0000 7f20ab59e640 1 -- 172.21.15.38:0/3410086811 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ac198210 con 0x7f20ac10b940 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.801+0000 7f20ab59e640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20ac10b940 0x7f20ac197a90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f20a400fe00 tx=0x7f20a400fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.801+0000 7f2093fff640 1 -- 172.21.15.38:0/3410086811 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a401d070 con 0x7f20ac10b940 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.801+0000 7f2093fff640 1 -- 172.21.15.38:0/3410086811 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20a4018920 con 0x7f20ac10b940 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.801+0000 7f20b1132640 1 -- 172.21.15.38:0/3410086811 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ac198500 con 0x7f20ac10b940 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.801+0000 7f2093fff640 1 -- 172.21.15.38:0/3410086811 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4007a80 con 0x7f20ac10b940 2024-03-26T06:39:20.806 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.801+0000 7f20b1132640 1 -- 172.21.15.38:0/3410086811 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ac100050 con 0x7f20ac10b940 2024-03-26T06:39:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.804+0000 7f2093fff640 1 -- 172.21.15.38:0/3410086811 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f20a4020070 con 0x7f20ac10b940 2024-03-26T06:39:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.804+0000 7f20b3dea740 1 Processor -- start 2024-03-26T06:39:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20b3dea740 1 -- start start 2024-03-26T06:39:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc6d40 0x555e83dc5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dc6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 0x555e83dd1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20b3dea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e83d2a5f0 con 0x555e83dc5fd0 2024-03-26T06:39:20.809 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20b3dea740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e83cd3930 con 0x555e83dc6d40 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20b3dea740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e83d26690 con 0x555e83dcf550 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20aad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dc6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20aad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dc6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46688/0 (socket says 172.21.15.38:46688) 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20aad9d640 1 -- 172.21.15.38:0/4278992778 learned_addr learned my addr 172.21.15.38:0/4278992778 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f2093fff640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f20940a2a10 0x7f20940a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20ab59e640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f20940a2a10 0x7f20940a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20aad9d640 1 -- 172.21.15.38:0/4278992778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 msgr2=0x555e83dd1940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20aad9d640 1 --2- 172.21.15.38:0/4278992778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 0x555e83dd1940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20aad9d640 1 -- 172.21.15.38:0/4278992778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc6d40 msgr2=0x555e83dc5a90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20aad9d640 1 --2- 172.21.15.38:0/4278992778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc6d40 0x555e83dc5a90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f20aad9d640 1 -- 172.21.15.38:0/4278992778 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e83a3f210 con 0x555e83dc5fd0 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.805+0000 7f2093fff640 1 -- 172.21.15.38:0/3410086811 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f20a40d2970 con 0x7f20ac10b940 2024-03-26T06:39:20.810 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.806+0000 7f20aad9d640 1 --2- 172.21.15.38:0/4278992778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dc6450 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f209c036040 tx=0x7f209c005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad3204d0a90837d server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.806+0000 7f20ab59e640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f20940a2a10 0x7f20940a4ed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f20a4010f50 tx=0x7f20a4070ca0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.806+0000 7f20917fa640 1 -- 172.21.15.38:0/4278992778 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c042020 con 0x555e83dc5fd0 2024-03-26T06:39:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.806+0000 7f20917fa640 1 -- 172.21.15.38:0/4278992778 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f209c02caa0 con 0x555e83dc5fd0 2024-03-26T06:39:20.811 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.806+0000 7f20917fa640 1 -- 172.21.15.38:0/4278992778 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c0046c0 con 0x555e83dc5fd0 2024-03-26T06:39:20.814 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.807+0000 7f20b3dea740 1 -- 172.21.15.38:0/4278992778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 msgr2=0x555e83dc6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:20.814 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.807+0000 7f20b3dea740 1 --2- 172.21.15.38:0/4278992778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dc6450 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f209c036040 tx=0x7f209c005f30 comp rx=0 tx=0).stop 2024-03-26T06:39:20.814 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.807+0000 7f20b3dea740 1 -- 172.21.15.38:0/4278992778 shutdown_connections 2024-03-26T06:39:20.814 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.807+0000 7f20b3dea740 1 --2- 172.21.15.38:0/4278992778 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 0x555e83dd1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.814 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.807+0000 7f20b3dea740 1 --2- 172.21.15.38:0/4278992778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dc6450 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.814 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.807+0000 7f20b3dea740 1 --2- 172.21.15.38:0/4278992778 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc6d40 0x555e83dc5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.814 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.807+0000 7f20b3dea740 1 -- 172.21.15.38:0/4278992778 >> 172.21.15.38:0/4278992778 conn(0x555e83d8d5b0 msgr2=0x555e83dc0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.807+0000 7f20b3dea740 1 -- 172.21.15.38:0/4278992778 shutdown_connections 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.807+0000 7f20b3dea740 1 -- 172.21.15.38:0/4278992778 wait complete. 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20b3dea740 1 -- start start 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20b3dea740 1 Processor -- start 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc5fd0 0x555e83dcc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc6d40 0x555e83dccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 0x555e83dcd060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20b3dea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e83dd2220 con 0x555e83dc6d40 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20b3dea740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e83ce6d90 con 0x555e83dc5fd0 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20b3dea740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e83dd23a0 con 0x555e83dcf550 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20b3dea740 10 client.0 _ll_register_callbacks cb 0x555e83a3ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20aad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc6d40 0x555e83dccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:20.815 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20aad9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc6d40 0x555e83dccb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46690/0 (socket says 172.21.15.38:46690) 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.808+0000 7f20aad9d640 1 -- 172.21.15.38:0/1291121148 learned_addr learned my addr 172.21.15.38:0/1291121148 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.809+0000 7f20aad9d640 1 -- 172.21.15.38:0/1291121148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 msgr2=0x555e83dcd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.809+0000 7f20aad9d640 1 --2- 172.21.15.38:0/1291121148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 0x555e83dcd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.809+0000 7f20aad9d640 1 -- 172.21.15.38:0/1291121148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc5fd0 msgr2=0x555e83dcc5e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.809+0000 7f20aad9d640 1 --2- 172.21.15.38:0/1291121148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc5fd0 0x555e83dcc5e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.809+0000 7f20aad9d640 1 -- 172.21.15.38:0/1291121148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e83b0b620 con 0x555e83dc6d40 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.809+0000 7f20aad9d640 1 --2- 172.21.15.38:0/1291121148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc6d40 0x555e83dccb20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f209c009400 tx=0x7f209c005220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d36b3fdcfed5dd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.809+0000 7f208eff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.809+0000 7f208eff5640 1 -- 172.21.15.38:0/1291121148 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209c004b50 con 0x555e83dc6d40 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.809+0000 7f20b3dea740 1 -- 172.21.15.38:0/1291121148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f209c02caa0 con 0x555e83dc6d40 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.810+0000 7f208eff5640 1 -- 172.21.15.38:0/1291121148 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f209c004cf0 con 0x555e83dc6d40 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.810+0000 7f208eff5640 1 -- 172.21.15.38:0/1291121148 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f209c03e460 con 0x555e83dc6d40 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.810+0000 7f20b3dea740 10 client.15534 fetch_fsmap learned FSMap version 27 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.810+0000 7f20b3dea740 1 -- 172.21.15.38:0/1291121148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555e83dde560 con 0x555e83dc6d40 2024-03-26T06:39:20.816 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.810+0000 7f208eff5640 1 -- 172.21.15.38:0/1291121148 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f209c02caa0 con 0x555e83dc6d40 2024-03-26T06:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.810+0000 7f208eff5640 1 -- 172.21.15.38:0/1291121148 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f209c0348b0 con 0x555e83dc6d40 2024-03-26T06:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.811+0000 7f20b3dea740 10 client.15534 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.811+0000 7f20b3dea740 10 client.15534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.811+0000 7f20b3dea740 20 client.15534 populate_metadata read hostname 'smithi038' 2024-03-26T06:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.811+0000 7f20b3dea740 1 --2- 172.21.15.38:0/1291121148 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x555e83ddf8b0 0x555e83de1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.811+0000 7f20b3dea740 4 client.15534 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:39:20.817 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.811+0000 7f20b3dea740 1 -- 172.21.15.38:0/1291121148 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e83d37b30 con 0x555e83ddf8b0 2024-03-26T06:39:20.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.812+0000 7f20aa59c640 1 --2- 172.21.15.38:0/1291121148 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x555e83ddf8b0 0x555e83de1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:20.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.815+0000 7f208eff5640 10 client.15534 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:20.818 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.815+0000 7f20aa59c640 1 --2- 172.21.15.38:0/1291121148 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x555e83ddf8b0 0x555e83de1ca0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=687e6f14e7658e80 server_cookie=e769a578743a3dd7 in_seq=0 out_seq=0 2024-03-26T06:39:20.822 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f208eff5640 1 -- 172.21.15.38:0/1291121148 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555e83d37b30 con 0x555e83ddf8b0 2024-03-26T06:39:20.822 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f208eff5640 10 client.15534 handle_command_reply: tid=0 2024-03-26T06:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f20b3dea740 1 client.15534 shutdown 2024-03-26T06:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f20b3dea740 1 -- 172.21.15.38:0/1291121148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc6d40 msgr2=0x555e83dccb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f20b3dea740 1 --2- 172.21.15.38:0/1291121148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc6d40 0x555e83dccb20 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f209c009400 tx=0x7f209c005220 comp rx=0 tx=0).stop 2024-03-26T06:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f20b3dea740 1 -- 172.21.15.38:0/1291121148 shutdown_connections 2024-03-26T06:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f20b3dea740 1 --2- 172.21.15.38:0/1291121148 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x555e83ddf8b0 0x555e83de1ca0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f20b3dea740 1 --2- 172.21.15.38:0/1291121148 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 0x555e83dcd060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f20b3dea740 1 --2- 172.21.15.38:0/1291121148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc6d40 0x555e83dccb20 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f208eff5640 0 client.15534 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f20b3dea740 1 --2- 172.21.15.38:0/1291121148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc5fd0 0x555e83dcc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.819+0000 7f20b3dea740 1 -- 172.21.15.38:0/1291121148 >> 172.21.15.38:0/1291121148 conn(0x555e83d8d5b0 msgr2=0x555e83dc2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:20.823 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.820+0000 7f20b3dea740 1 -- 172.21.15.38:0/1291121148 shutdown_connections 2024-03-26T06:39:20.824 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.820+0000 7f20b3dea740 1 -- 172.21.15.38:0/1291121148 wait complete. 2024-03-26T06:39:20.824 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.820+0000 7f20b3dea740 20 client.15534 trim_cache size 0 max 16384 2024-03-26T06:39:20.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:20 smithi038 ceph-mon[32265]: pgmap v57: 97 pgs: 97 active+clean; 6.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 586 MiB/s wr, 408 op/s 2024-03-26T06:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20b3dea740 1 Processor -- start 2024-03-26T06:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20b3dea740 1 -- start start 2024-03-26T06:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc6d40 0x555e83dd6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 0x555e83dd6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20b3dea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac1ad140 con 0x555e83dc5fd0 2024-03-26T06:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20b3dea740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac1aac80 con 0x555e83dc6d40 2024-03-26T06:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20b3dea740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac1a8d40 con 0x555e83dcf550 2024-03-26T06:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20aa59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20aa59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46706/0 (socket says 172.21.15.38:46706) 2024-03-26T06:39:20.844 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20aa59c640 1 -- 172.21.15.38:0/3159609784 learned_addr learned my addr 172.21.15.38:0/3159609784 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20aa59c640 1 -- 172.21.15.38:0/3159609784 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 msgr2=0x555e83dd6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20aa59c640 1 --2- 172.21.15.38:0/3159609784 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 0x555e83dd6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.841+0000 7f20aa59c640 1 -- 172.21.15.38:0/3159609784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc6d40 msgr2=0x555e83dd6610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-26T06:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.842+0000 7f20aa59c640 1 --2- 172.21.15.38:0/3159609784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc6d40 0x555e83dd6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.842+0000 7f20aa59c640 1 -- 172.21.15.38:0/3159609784 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e83dcab30 con 0x555e83dc5fd0 2024-03-26T06:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.842+0000 7f20aa59c640 1 --2- 172.21.15.38:0/3159609784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dced10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009500 tx=0x7f20a0009a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0cf0e2a501d3e76 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:20.847 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.842+0000 7f2090ff9640 1 -- 172.21.15.38:0/3159609784 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a001d670 con 0x555e83dc5fd0 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.842+0000 7f2090ff9640 1 -- 172.21.15.38:0/3159609784 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20a0018070 con 0x555e83dc5fd0 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.842+0000 7f2090ff9640 1 -- 172.21.15.38:0/3159609784 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0011b40 con 0x555e83dc5fd0 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.843+0000 7f20b3dea740 1 -- 172.21.15.38:0/3159609784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 msgr2=0x555e83dced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.843+0000 7f20b3dea740 1 --2- 172.21.15.38:0/3159609784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dced10 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009500 tx=0x7f20a0009a60 comp rx=0 tx=0).stop 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.843+0000 7f20b3dea740 1 -- 172.21.15.38:0/3159609784 shutdown_connections 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.843+0000 7f20b3dea740 1 --2- 172.21.15.38:0/3159609784 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dcf550 0x555e83dd6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.843+0000 7f20b3dea740 1 --2- 172.21.15.38:0/3159609784 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc6d40 0x555e83dd6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.843+0000 7f20b3dea740 1 --2- 172.21.15.38:0/3159609784 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dc5fd0 0x555e83dced10 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.843+0000 7f20b3dea740 1 -- 172.21.15.38:0/3159609784 >> 172.21.15.38:0/3159609784 conn(0x555e83d8d5b0 msgr2=0x555e83dbbd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.843+0000 7f20b3dea740 1 -- 172.21.15.38:0/3159609784 shutdown_connections 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.843+0000 7f20b3dea740 1 -- 172.21.15.38:0/3159609784 wait complete. 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.843+0000 7f20b3dea740 1 -- start start 2024-03-26T06:39:20.848 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.843+0000 7f20b3dea740 1 Processor -- start 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc5fd0 0x555e83dd6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dc6d40 0x555e83dc8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20b3dea740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dcf550 0x555e83dc9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20b3dea740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac1a9340 con 0x555e83dcf550 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20b3dea740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac1aa620 con 0x555e83dc5fd0 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20b3dea740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ac1a9960 con 0x555e83dc6d40 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20b3dea740 10 client.0 _ll_register_callbacks cb 0x7f209c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20aa59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dcf550 0x555e83dc9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20aa59c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dcf550 0x555e83dc9160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46718/0 (socket says 172.21.15.38:46718) 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20aa59c640 1 -- 172.21.15.38:0/3924479247 learned_addr learned my addr 172.21.15.38:0/3924479247 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20aa59c640 1 -- 172.21.15.38:0/3924479247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dc6d40 msgr2=0x555e83dc8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20aa59c640 1 --2- 172.21.15.38:0/3924479247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dc6d40 0x555e83dc8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20aa59c640 1 -- 172.21.15.38:0/3924479247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc5fd0 msgr2=0x555e83dd6620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-26T06:39:20.849 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20aa59c640 1 --2- 172.21.15.38:0/3924479247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc5fd0 0x555e83dd6620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.844+0000 7f20aa59c640 1 -- 172.21.15.38:0/3924479247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e83dc96a0 con 0x555e83dcf550 2024-03-26T06:39:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.845+0000 7f20aa59c640 1 --2- 172.21.15.38:0/3924479247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dcf550 0x555e83dc9160 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f20a00115c0 tx=0x7f20a00048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ad01cf49f12673 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:20.850 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.845+0000 7f208eff5640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:39:20.851 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.845+0000 7f208eff5640 1 -- 172.21.15.38:0/3924479247 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a0011d90 con 0x555e83dcf550 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.845+0000 7f208eff5640 1 -- 172.21.15.38:0/3924479247 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f20a001d470 con 0x555e83dcf550 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.845+0000 7f20b3dea740 1 -- 172.21.15.38:0/3924479247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f20a0018070 con 0x555e83dcf550 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.845+0000 7f208eff5640 1 -- 172.21.15.38:0/3924479247 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20a001d670 con 0x555e83dcf550 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.845+0000 7f20b3dea740 10 client.15540 fetch_fsmap learned FSMap version 27 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.845+0000 7f20b3dea740 1 -- 172.21.15.38:0/3924479247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x555e83e3eff0 con 0x555e83dcf550 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.846+0000 7f208eff5640 1 -- 172.21.15.38:0/3924479247 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f20a0018070 con 0x555e83dcf550 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.846+0000 7f208eff5640 1 -- 172.21.15.38:0/3924479247 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f20a0024030 con 0x555e83dcf550 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.846+0000 7f20b3dea740 10 client.15540 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.846+0000 7f20b3dea740 10 client.15540 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.846+0000 7f20b3dea740 20 client.15540 populate_metadata read hostname 'smithi038' 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.846+0000 7f20b3dea740 1 --2- 172.21.15.38:0/3924479247 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x555e83e3f640 0x555e83e41a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.846+0000 7f20b3dea740 4 client.15540 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:39:20.852 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.846+0000 7f20b3dea740 1 -- 172.21.15.38:0/3924479247 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555e83d31e80 con 0x555e83e3f640 2024-03-26T06:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.847+0000 7f20aad9d640 1 --2- 172.21.15.38:0/3924479247 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x555e83e3f640 0x555e83e41a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.848+0000 7f208eff5640 10 client.15540 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.847+0000 7f20aad9d640 1 --2- 172.21.15.38:0/3924479247 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x555e83e3f640 0x555e83e41a60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75f9e4457a0d4dd9 server_cookie=9fab3d71c7f38847 in_seq=0 out_seq=0 2024-03-26T06:39:20.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:20 smithi121 ceph-mon[36349]: pgmap v57: 97 pgs: 97 active+clean; 6.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 586 MiB/s wr, 408 op/s 2024-03-26T06:39:20.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:20 smithi170 ceph-mon[36912]: pgmap v57: 97 pgs: 97 active+clean; 6.9 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 586 MiB/s wr, 408 op/s 2024-03-26T06:39:20.976 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.972+0000 7f208eff5640 1 -- 172.21.15.38:0/3924479247 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555e83d31e80 con 0x555e83e3f640 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.972+0000 7f208eff5640 10 client.15540 handle_command_reply: tid=0 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.972+0000 7f20b3dea740 1 client.15540 shutdown 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.972+0000 7f20b3dea740 1 -- 172.21.15.38:0/3924479247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dcf550 msgr2=0x555e83dc9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.973+0000 7f20b3dea740 1 --2- 172.21.15.38:0/3924479247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dcf550 0x555e83dc9160 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f20a00115c0 tx=0x7f20a00048b0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.973+0000 7f20b3dea740 1 -- 172.21.15.38:0/3924479247 shutdown_connections 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.973+0000 7f20b3dea740 1 --2- 172.21.15.38:0/3924479247 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x555e83e3f640 0x555e83e41a60 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.973+0000 7f20b3dea740 1 --2- 172.21.15.38:0/3924479247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e83dcf550 0x555e83dc9160 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.973+0000 7f208eff5640 0 client.15540 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.973+0000 7f20b3dea740 1 --2- 172.21.15.38:0/3924479247 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x555e83dc6d40 0x555e83dc8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.973+0000 7f20b3dea740 1 --2- 172.21.15.38:0/3924479247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x555e83dc5fd0 0x555e83dd6620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.973+0000 7f20b3dea740 1 -- 172.21.15.38:0/3924479247 >> 172.21.15.38:0/3924479247 conn(0x555e83d8d5b0 msgr2=0x555e83dc07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.973+0000 7f20b3dea740 1 -- 172.21.15.38:0/3924479247 shutdown_connections 2024-03-26T06:39:20.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.974+0000 7f20b3dea740 1 -- 172.21.15.38:0/3924479247 wait complete. 2024-03-26T06:39:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.974+0000 7f20b3dea740 20 client.15540 trim_cache size 0 max 16384 2024-03-26T06:39:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:39:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:39:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:39:20.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:39:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.976+0000 7f20b1132640 1 -- 172.21.15.38:0/3410086811 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f20940a2a10 msgr2=0x7f20940a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:20.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.976+0000 7f20b1132640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f20940a2a10 0x7f20940a4ed0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f20a4010f50 tx=0x7f20a4070ca0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.976+0000 7f20b1132640 1 -- 172.21.15.38:0/3410086811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20ac10b940 msgr2=0x7f20ac197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:20.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.977+0000 7f20b1132640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20ac10b940 0x7f20ac197a90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f20a400fe00 tx=0x7f20a400fef0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.977+0000 7f20b1132640 1 -- 172.21.15.38:0/3410086811 shutdown_connections 2024-03-26T06:39:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.978+0000 7f20b1132640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f20940a2a10 0x7f20940a4ed0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.978+0000 7f20b1132640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f20ac10b940 0x7f20ac197a90 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.978+0000 7f20b1132640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f20ac102df0 0x7f20ac19ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.978+0000 7f20b1132640 1 --2- 172.21.15.38:0/3410086811 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f20ac1024d0 0x7f20ac19d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:20.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.978+0000 7f20b1132640 1 -- 172.21.15.38:0/3410086811 >> 172.21.15.38:0/3410086811 conn(0x7f20ac077320 msgr2=0x7f20ac0fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:20.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.982+0000 7f20b1132640 1 -- 172.21.15.38:0/3410086811 shutdown_connections 2024-03-26T06:39:20.985 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:20.982+0000 7f20b1132640 1 -- 172.21.15.38:0/3410086811 wait complete. 2024-03-26T06:39:21.889 INFO:tasks.workunit.client.0.smithi038.stdout:fs setup took 15 secs 2024-03-26T06:39:22.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:22 smithi038 ceph-mon[32265]: pgmap v58: 97 pgs: 97 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 595 MiB/s wr, 247 op/s 2024-03-26T06:39:22.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:22 smithi121 ceph-mon[36349]: pgmap v58: 97 pgs: 97 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 595 MiB/s wr, 247 op/s 2024-03-26T06:39:22.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:22 smithi170 ceph-mon[36912]: pgmap v58: 97 pgs: 97 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 595 MiB/s wr, 247 op/s 2024-03-26T06:39:23.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:23 smithi038 ceph-mon[32265]: pgmap v59: 97 pgs: 97 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 595 MiB/s wr, 247 op/s 2024-03-26T06:39:23.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:23 smithi121 ceph-mon[36349]: pgmap v59: 97 pgs: 97 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 595 MiB/s wr, 247 op/s 2024-03-26T06:39:23.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:23 smithi170 ceph-mon[36912]: pgmap v59: 97 pgs: 97 active+clean; 7.3 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 595 MiB/s wr, 247 op/s 2024-03-26T06:39:25.230 INFO:tasks.workunit.client.0.smithi038.stdout:Syncing()...3 sec 2024-03-26T06:39:25.230 INFO:tasks.workunit.client.0.smithi038.stdout:Starting Actual Benchmark At: Tue Mar 26 06:39:25 2024 2024-03-26T06:39:25.231 INFO:tasks.workunit.client.0.smithi038.stdout: 2024-03-26T06:39:26.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:25 smithi121 ceph-mon[36349]: pgmap v60: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 936 MiB/s wr, 381 op/s 2024-03-26T06:39:26.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:25 smithi170 ceph-mon[36912]: pgmap v60: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 936 MiB/s wr, 381 op/s 2024-03-26T06:39:26.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:25 smithi038 ceph-mon[32265]: pgmap v60: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 936 MiB/s wr, 381 op/s 2024-03-26T06:39:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:27 smithi121 ceph-mon[36349]: pgmap v61: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 643 MiB/s wr, 247 op/s 2024-03-26T06:39:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:27 smithi170 ceph-mon[36912]: pgmap v61: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 643 MiB/s wr, 247 op/s 2024-03-26T06:39:28.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:27 smithi038 ceph-mon[32265]: pgmap v61: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 643 MiB/s wr, 247 op/s 2024-03-26T06:39:30.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:29 smithi121 ceph-mon[36349]: pgmap v62: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 643 MiB/s wr, 247 op/s 2024-03-26T06:39:30.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:29 smithi170 ceph-mon[36912]: pgmap v62: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 643 MiB/s wr, 247 op/s 2024-03-26T06:39:30.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:29 smithi038 ceph-mon[32265]: pgmap v62: 97 pgs: 97 active+clean; 11 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 643 MiB/s wr, 247 op/s 2024-03-26T06:39:32.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:31 smithi121 ceph-mon[36349]: pgmap v63: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 541 MiB/s wr, 9.24k op/s 2024-03-26T06:39:32.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:31 smithi170 ceph-mon[36912]: pgmap v63: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 541 MiB/s wr, 9.24k op/s 2024-03-26T06:39:32.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:31 smithi038 ceph-mon[32265]: pgmap v63: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 541 MiB/s wr, 9.24k op/s 2024-03-26T06:39:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.150+0000 7fb58da97640 1 -- 172.21.15.38:0/1590486986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb58811aad0 msgr2=0x7fb58811cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.150+0000 7fb5867fc640 1 -- 172.21.15.38:0/1590486986 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb580039070 con 0x7fb58811aad0 2024-03-26T06:39:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.150+0000 7fb58da97640 1 --2- 172.21.15.38:0/1590486986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb58811aad0 0x7fb58811cec0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb580009ca0 tx=0x7fb58002c140 comp rx=0 tx=0).stop 2024-03-26T06:39:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.150+0000 7fb58da97640 1 -- 172.21.15.38:0/1590486986 shutdown_connections 2024-03-26T06:39:34.153 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.150+0000 7fb58da97640 1 --2- 172.21.15.38:0/1590486986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb58811aad0 0x7fb58811cec0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.150+0000 7fb58da97640 1 --2- 172.21.15.38:0/1590486986 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb58810fea0 0x7fb58810db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.150+0000 7fb58da97640 1 --2- 172.21.15.38:0/1590486986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb58810fa30 0x7fb58810d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.150+0000 7fb58da97640 1 -- 172.21.15.38:0/1590486986 >> 172.21.15.38:0/1590486986 conn(0x7fb58806c390 msgr2=0x7fb58806c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.151+0000 7fb58da97640 1 -- 172.21.15.38:0/1590486986 shutdown_connections 2024-03-26T06:39:34.154 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.152+0000 7fb58da97640 1 -- 172.21.15.38:0/1590486986 wait complete. 2024-03-26T06:39:34.155 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.153+0000 7fb58da97640 1 Processor -- start 2024-03-26T06:39:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.153+0000 7fb58da97640 1 -- start start 2024-03-26T06:39:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.153+0000 7fb58da97640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb58810fa30 0x7fb5881ac610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.153+0000 7fb58da97640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb58810fea0 0x7fb5881acb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.153+0000 7fb58da97640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb58811aad0 0x7fb5881a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.153+0000 7fb58da97640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb588120560 con 0x7fb58811aad0 2024-03-26T06:39:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.153+0000 7fb58da97640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5881203e0 con 0x7fb58810fa30 2024-03-26T06:39:34.157 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.153+0000 7fb58da97640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5881206e0 con 0x7fb58810fea0 2024-03-26T06:39:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.154+0000 7fb586ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb58810fea0 0x7fb5881acb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.154+0000 7fb586ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb58810fea0 0x7fb5881acb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:41030/0 (socket says 172.21.15.38:41030) 2024-03-26T06:39:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.154+0000 7fb586ffd640 1 -- 172.21.15.38:0/4230339581 learned_addr learned my addr 172.21.15.38:0/4230339581 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.154+0000 7fb587fff640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb58811aad0 0x7fb5881a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.155+0000 7fb586ffd640 1 -- 172.21.15.38:0/4230339581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb58810fa30 msgr2=0x7fb5881ac610 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-26T06:39:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.155+0000 7fb586ffd640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb58810fa30 0x7fb5881ac610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.155+0000 7fb586ffd640 1 -- 172.21.15.38:0/4230339581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb58811aad0 msgr2=0x7fb5881a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.155+0000 7fb586ffd640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb58811aad0 0x7fb5881a6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.158 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.155+0000 7fb586ffd640 1 -- 172.21.15.38:0/4230339581 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5881a6f50 con 0x7fb58810fea0 2024-03-26T06:39:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.155+0000 7fb586ffd640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb58810fea0 0x7fb5881acb50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00d770 tx=0x7fb57c00dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.155+0000 7fb584ff9640 1 -- 172.21.15.38:0/4230339581 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c017070 con 0x7fb58810fea0 2024-03-26T06:39:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.155+0000 7fb58da97640 1 -- 172.21.15.38:0/4230339581 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5881a7240 con 0x7fb58810fea0 2024-03-26T06:39:34.159 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.156+0000 7fb58da97640 1 -- 172.21.15.38:0/4230339581 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5881102b0 con 0x7fb58810fea0 2024-03-26T06:39:34.163 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.160+0000 7fb590677740 1 Processor -- start 2024-03-26T06:39:34.163 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.160+0000 7fb584ff9640 1 -- 172.21.15.38:0/4230339581 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb57c009c30 con 0x7fb58810fea0 2024-03-26T06:39:34.163 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.160+0000 7fb584ff9640 1 -- 172.21.15.38:0/4230339581 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c00f620 con 0x7fb58810fea0 2024-03-26T06:39:34.164 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.162+0000 7fb590677740 1 -- start start 2024-03-26T06:39:34.165 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.163+0000 7fb590677740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f9195bd40 0x558f9195aa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.163+0000 7fb590677740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195afd0 0x558f9195b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.163+0000 7fb590677740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 0x558f91966940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.163+0000 7fb590677740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f918bf5f0 con 0x558f9195afd0 2024-03-26T06:39:34.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.163+0000 7fb590677740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f91868930 con 0x558f91964550 2024-03-26T06:39:34.166 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.163+0000 7fb590677740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f918bb690 con 0x558f9195bd40 2024-03-26T06:39:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.164+0000 7fb587fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f9195bd40 0x558f9195aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.165+0000 7fb587fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f9195bd40 0x558f9195aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:41044/0 (socket says 172.21.15.38:41044) 2024-03-26T06:39:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.165+0000 7fb587fff640 1 -- 172.21.15.38:0/3762742225 learned_addr learned my addr 172.21.15.38:0/3762742225 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.165+0000 7fb586ffd640 1 --2- 172.21.15.38:0/3762742225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 0x558f91966940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.168 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.165+0000 7fb584ff9640 1 -- 172.21.15.38:0/4230339581 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb57c01a070 con 0x7fb58810fea0 2024-03-26T06:39:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.166+0000 7fb584ff9640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb5700a2b60 0x7fb5700a5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.169 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.166+0000 7fb584ff9640 1 -- 172.21.15.38:0/4230339581 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fb57c0ced30 con 0x7fb58810fea0 2024-03-26T06:39:34.170 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.167+0000 7fb5877fe640 1 --2- 172.21.15.38:0/3762742225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195afd0 0x558f9195b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.168+0000 7fb5877fe640 1 -- 172.21.15.38:0/3762742225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f9195bd40 msgr2=0x558f9195aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.168+0000 7fb5877fe640 1 --2- 172.21.15.38:0/3762742225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f9195bd40 0x558f9195aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.168+0000 7fb5877fe640 1 -- 172.21.15.38:0/3762742225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 msgr2=0x558f91966940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.168+0000 7fb5877fe640 1 --2- 172.21.15.38:0/3762742225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 0x558f91966940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.168+0000 7fb5877fe640 1 -- 172.21.15.38:0/3762742225 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f915d4210 con 0x558f9195afd0 2024-03-26T06:39:34.171 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.169+0000 7fb5877fe640 1 --2- 172.21.15.38:0/3762742225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195afd0 0x558f9195b450 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb578009690 tx=0x7fb578009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da3eb9f52f4dd1d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.169+0000 7fb56dffb640 1 -- 172.21.15.38:0/3762742225 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578015020 con 0x558f9195afd0 2024-03-26T06:39:34.172 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.169+0000 7fb56dffb640 1 -- 172.21.15.38:0/3762742225 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb57800bda0 con 0x558f9195afd0 2024-03-26T06:39:34.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.170+0000 7fb587fff640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb5700a2b60 0x7fb5700a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.170+0000 7fb590677740 1 -- 172.21.15.38:0/3762742225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195afd0 msgr2=0x558f9195b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:34.173 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.170+0000 7fb590677740 1 --2- 172.21.15.38:0/3762742225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195afd0 0x558f9195b450 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb578009690 tx=0x7fb578009b50 comp rx=0 tx=0).stop 2024-03-26T06:39:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.172+0000 7fb590677740 1 -- 172.21.15.38:0/3762742225 shutdown_connections 2024-03-26T06:39:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.172+0000 7fb590677740 1 --2- 172.21.15.38:0/3762742225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 0x558f91966940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.172+0000 7fb590677740 1 --2- 172.21.15.38:0/3762742225 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195afd0 0x558f9195b450 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.172+0000 7fb590677740 1 --2- 172.21.15.38:0/3762742225 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f9195bd40 0x558f9195aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.175 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.172+0000 7fb590677740 1 -- 172.21.15.38:0/3762742225 >> 172.21.15.38:0/3762742225 conn(0x558f919225b0 msgr2=0x558f91955cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:34.177 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.175+0000 7fb590677740 1 -- 172.21.15.38:0/3762742225 shutdown_connections 2024-03-26T06:39:34.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.178+0000 7fb590677740 1 -- 172.21.15.38:0/3762742225 wait complete. 2024-03-26T06:39:34.182 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.179+0000 7fb590677740 1 -- start start 2024-03-26T06:39:34.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.180+0000 7fb590677740 1 Processor -- start 2024-03-26T06:39:34.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.180+0000 7fb590677740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f9195bd40 0x558f91963f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.180+0000 7fb590677740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 0x558f91961210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.180+0000 7fb590677740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f919617a0 0x558f91961c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.180+0000 7fb590677740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f919672d0 con 0x558f919617a0 2024-03-26T06:39:34.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.180+0000 7fb590677740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9187bd90 con 0x558f91964550 2024-03-26T06:39:34.183 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.180+0000 7fb590677740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f91967450 con 0x558f9195bd40 2024-03-26T06:39:34.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.180+0000 7fb590677740 10 client.0 _ll_register_callbacks cb 0x558f915d3a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:39:34.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.181+0000 7fb586ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 0x558f91961210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.181+0000 7fb586ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 0x558f91961210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:51718/0 (socket says 172.21.15.38:51718) 2024-03-26T06:39:34.184 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.181+0000 7fb586ffd640 1 -- 172.21.15.38:0/505861083 learned_addr learned my addr 172.21.15.38:0/505861083 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:34.185 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.182+0000 7fb5877fe640 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f9195bd40 0x558f91963f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.182+0000 7fb586ffd640 1 -- 172.21.15.38:0/505861083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f9195bd40 msgr2=0x558f91963f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.182+0000 7fb586ffd640 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f9195bd40 0x558f91963f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.182+0000 7fb586ffd640 1 -- 172.21.15.38:0/505861083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f919617a0 msgr2=0x558f91961c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.182+0000 7fb586ffd640 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f919617a0 0x558f91961c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.182+0000 7fb586ffd640 1 -- 172.21.15.38:0/505861083 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f916a0620 con 0x558f91964550 2024-03-26T06:39:34.186 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.183+0000 7fb587fff640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb5700a2b60 0x7fb5700a5020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb580032520 tx=0x7fb58002c140 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:34.189 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.186+0000 7fb586ffd640 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 0x558f91961210 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb57c09b2b0 tx=0x7fb57c09bb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdff15bc93df21c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:34.189 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.186+0000 7fb566ffd640 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-03-26T06:39:34.192 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.189+0000 7fb566ffd640 1 -- 172.21.15.38:0/505861083 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c021070 con 0x558f91964550 2024-03-26T06:39:34.192 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.189+0000 7fb566ffd640 1 -- 172.21.15.38:0/505861083 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fb57c0cc690 con 0x558f91964550 2024-03-26T06:39:34.192 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.189+0000 7fb590677740 1 -- 172.21.15.38:0/505861083 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558f918ccb30 con 0x558f91964550 2024-03-26T06:39:34.192 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.190+0000 7fb566ffd640 1 -- 172.21.15.38:0/505861083 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb57c0cc890 con 0x558f91964550 2024-03-26T06:39:34.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.191+0000 7fb590677740 10 client.25163 fetch_fsmap learned FSMap version 27 2024-03-26T06:39:34.193 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.191+0000 7fb590677740 1 -- 172.21.15.38:0/505861083 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558f91973840 con 0x558f91964550 2024-03-26T06:39:34.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.191+0000 7fb566ffd640 1 -- 172.21.15.38:0/505861083 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558f918ccb30 con 0x558f91964550 2024-03-26T06:39:34.194 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.192+0000 7fb566ffd640 1 -- 172.21.15.38:0/505861083 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fb57c0a3630 con 0x558f91964550 2024-03-26T06:39:34.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.192+0000 7fb590677740 10 client.25163 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:39:34.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.192+0000 7fb590677740 10 client.25163 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:39:34.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.192+0000 7fb590677740 20 client.25163 populate_metadata read hostname 'smithi038' 2024-03-26T06:39:34.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.192+0000 7fb590677740 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x558f91974cb0 0x558f919770c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.192+0000 7fb590677740 4 client.25163 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:39:34.195 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.192+0000 7fb590677740 1 -- 172.21.15.38:0/505861083 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f918c6e80 con 0x558f91974cb0 2024-03-26T06:39:34.200 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.194+0000 7fb5877fe640 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x558f91974cb0 0x558f919770c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.206+0000 7fb566ffd640 10 client.25163 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:34.209 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.206+0000 7fb5877fe640 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x558f91974cb0 0x558f919770c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb19ab30980910cf server_cookie=dd16171a3638b06e in_seq=0 out_seq=0 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.212+0000 7fb566ffd640 1 -- 172.21.15.38:0/505861083 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558f918c6e80 con 0x558f91974cb0 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.212+0000 7fb566ffd640 10 client.25163 handle_command_reply: tid=0 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.212+0000 7fb590677740 1 client.25163 shutdown 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb590677740 1 -- 172.21.15.38:0/505861083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 msgr2=0x558f91961210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb590677740 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 0x558f91961210 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb57c09b2b0 tx=0x7fb57c09bb60 comp rx=0 tx=0).stop 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb590677740 1 -- 172.21.15.38:0/505861083 shutdown_connections 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb590677740 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x558f91974cb0 0x558f919770c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb590677740 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f919617a0 0x558f91961c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb590677740 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f91964550 0x558f91961210 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb590677740 1 --2- 172.21.15.38:0/505861083 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f9195bd40 0x558f91963f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb590677740 1 -- 172.21.15.38:0/505861083 >> 172.21.15.38:0/505861083 conn(0x558f919225b0 msgr2=0x558f91957180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:34.218 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb566ffd640 0 client.25163 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb590677740 1 -- 172.21.15.38:0/505861083 shutdown_connections 2024-03-26T06:39:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb590677740 1 -- 172.21.15.38:0/505861083 wait complete. 2024-03-26T06:39:34.219 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.213+0000 7fb590677740 20 client.25163 trim_cache size 0 max 16384 2024-03-26T06:39:34.257 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.254+0000 7fb590677740 1 Processor -- start 2024-03-26T06:39:34.260 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.258+0000 7fb590677740 1 -- start start 2024-03-26T06:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.258+0000 7fb590677740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195bd40 0x558f91965940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.259+0000 7fb590677740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f919617a0 0x558f91963a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.259+0000 7fb590677740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f91964550 0x558f9195fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.259+0000 7fb590677740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb574010c90 con 0x558f9195bd40 2024-03-26T06:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.259+0000 7fb586ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f919617a0 0x558f91963a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.259+0000 7fb586ffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f919617a0 0x558f91963a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:51722/0 (socket says 172.21.15.38:51722) 2024-03-26T06:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.259+0000 7fb586ffd640 1 -- 172.21.15.38:0/1090137037 learned_addr learned my addr 172.21.15.38:0/1090137037 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.259+0000 7fb590677740 1 -- 172.21.15.38:0/1090137037 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57400e7d0 con 0x558f919617a0 2024-03-26T06:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.259+0000 7fb587fff640 1 --2- 172.21.15.38:0/1090137037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f91964550 0x558f9195fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.259+0000 7fb590677740 1 -- 172.21.15.38:0/1090137037 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57400c870 con 0x558f91964550 2024-03-26T06:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.260+0000 7fb5877fe640 1 --2- 172.21.15.38:0/1090137037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195bd40 0x558f91965940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.261+0000 7fb5877fe640 1 -- 172.21.15.38:0/1090137037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f91964550 msgr2=0x558f9195fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.261+0000 7fb5877fe640 1 --2- 172.21.15.38:0/1090137037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f91964550 0x558f9195fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.261+0000 7fb5877fe640 1 -- 172.21.15.38:0/1090137037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f919617a0 msgr2=0x558f91963a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.261+0000 7fb5877fe640 1 --2- 172.21.15.38:0/1090137037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f919617a0 0x558f91963a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.264 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.261+0000 7fb5877fe640 1 -- 172.21.15.38:0/1090137037 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f9195a2a0 con 0x558f9195bd40 2024-03-26T06:39:34.267 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.265+0000 7fb5877fe640 1 --2- 172.21.15.38:0/1090137037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195bd40 0x558f91965940 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb578006fd0 tx=0x7fb578009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6697f2abe91bdb33 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.265+0000 7fb56d7fa640 1 -- 172.21.15.38:0/1090137037 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578028070 con 0x558f9195bd40 2024-03-26T06:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.266+0000 7fb56d7fa640 1 -- 172.21.15.38:0/1090137037 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb578023d80 con 0x558f9195bd40 2024-03-26T06:39:34.268 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.266+0000 7fb56d7fa640 1 -- 172.21.15.38:0/1090137037 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578015040 con 0x558f9195bd40 2024-03-26T06:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.267+0000 7fb590677740 1 -- 172.21.15.38:0/1090137037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195bd40 msgr2=0x558f91965940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.267+0000 7fb590677740 1 --2- 172.21.15.38:0/1090137037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195bd40 0x558f91965940 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb578006fd0 tx=0x7fb578009060 comp rx=0 tx=0).stop 2024-03-26T06:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.267+0000 7fb590677740 1 -- 172.21.15.38:0/1090137037 shutdown_connections 2024-03-26T06:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.267+0000 7fb590677740 1 --2- 172.21.15.38:0/1090137037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f91964550 0x558f9195fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.267+0000 7fb590677740 1 --2- 172.21.15.38:0/1090137037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f919617a0 0x558f91963a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.270 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.267+0000 7fb590677740 1 --2- 172.21.15.38:0/1090137037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f9195bd40 0x558f91965940 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.267+0000 7fb590677740 1 -- 172.21.15.38:0/1090137037 >> 172.21.15.38:0/1090137037 conn(0x558f919225b0 msgr2=0x558f9195c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.267+0000 7fb590677740 1 -- 172.21.15.38:0/1090137037 shutdown_connections 2024-03-26T06:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.268+0000 7fb590677740 1 -- 172.21.15.38:0/1090137037 wait complete. 2024-03-26T06:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.268+0000 7fb590677740 1 -- start start 2024-03-26T06:39:34.271 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.269+0000 7fb590677740 1 Processor -- start 2024-03-26T06:39:34.272 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.269+0000 7fb590677740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f9195bd40 0x558f9196b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.273 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.269+0000 7fb590677740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f919617a0 0x558f9196bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.273 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.270+0000 7fb590677740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f91964550 0x558f9195d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.273 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.270+0000 7fb586ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f919617a0 0x558f9196bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.273 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.270+0000 7fb586ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f919617a0 0x558f9196bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:41066/0 (socket says 172.21.15.38:41066) 2024-03-26T06:39:34.273 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.270+0000 7fb5877fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f9195bd40 0x558f9196b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.273 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.270+0000 7fb5877fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f9195bd40 0x558f9196b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:51736/0 (socket says 172.21.15.38:51736) 2024-03-26T06:39:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.270+0000 7fb586ffd640 1 -- 172.21.15.38:0/3891901569 learned_addr learned my addr 172.21.15.38:0/3891901569 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.270+0000 7fb590677740 1 -- 172.21.15.38:0/3891901569 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57400ce70 con 0x558f91964550 2024-03-26T06:39:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.270+0000 7fb590677740 1 -- 172.21.15.38:0/3891901569 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57400e170 con 0x558f9195bd40 2024-03-26T06:39:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.270+0000 7fb590677740 1 -- 172.21.15.38:0/3891901569 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57400d4b0 con 0x558f919617a0 2024-03-26T06:39:34.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.270+0000 7fb590677740 10 client.0 _ll_register_callbacks cb 0x7fb578006e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:39:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.271+0000 7fb587fff640 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f91964550 0x558f9195d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.271+0000 7fb586ffd640 1 -- 172.21.15.38:0/3891901569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f9195bd40 msgr2=0x558f9196b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.271+0000 7fb586ffd640 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f9195bd40 0x558f9196b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.271+0000 7fb586ffd640 1 -- 172.21.15.38:0/3891901569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f91964550 msgr2=0x558f9195d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.271+0000 7fb586ffd640 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f91964550 0x558f9195d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.271+0000 7fb586ffd640 1 -- 172.21.15.38:0/3891901569 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558f9195dc10 con 0x558f919617a0 2024-03-26T06:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.272+0000 7fb590677740 1 -- 172.21.15.38:0/3891901569 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb578023d80 con 0x558f919617a0 2024-03-26T06:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.272+0000 7fb586ffd640 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f919617a0 0x558f9196bc90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb57c09b2b0 tx=0x7fb57c097ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b25c7910e0b1760f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.272+0000 7fb566ffd640 10 client.25066 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-26T06:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.272+0000 7fb566ffd640 1 -- 172.21.15.38:0/3891901569 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c09ac80 con 0x558f919617a0 2024-03-26T06:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.272+0000 7fb566ffd640 1 -- 172.21.15.38:0/3891901569 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fb57c0cc690 con 0x558f919617a0 2024-03-26T06:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.272+0000 7fb566ffd640 1 -- 172.21.15.38:0/3891901569 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb57c09b0f0 con 0x558f919617a0 2024-03-26T06:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.273+0000 7fb590677740 10 client.25066 fetch_fsmap learned FSMap version 27 2024-03-26T06:39:34.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.273+0000 7fb590677740 1 -- 172.21.15.38:0/3891901569 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({fsmap=27}) v3 -- 0x558f919cba90 con 0x558f919617a0 2024-03-26T06:39:34.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.274+0000 7fb566ffd640 1 -- 172.21.15.38:0/3891901569 <== mon.1 v2:172.21.15.170:3300/0 4 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fb57c09a770 con 0x558f919617a0 2024-03-26T06:39:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.274+0000 7fb590677740 10 client.25066 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:39:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.274+0000 7fb590677740 10 client.25066 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:39:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.274+0000 7fb590677740 20 client.25066 populate_metadata read hostname 'smithi038' 2024-03-26T06:39:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.274+0000 7fb590677740 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x558f919cbfb0 0x558f919ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.274+0000 7fb590677740 4 client.25066 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:39:34.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.274+0000 7fb590677740 1 -- 172.21.15.38:0/3891901569 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7fb578023d80 con 0x558f919cbfb0 2024-03-26T06:39:34.279 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.277+0000 7fb5877fe640 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x558f919cbfb0 0x558f919ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.290+0000 7fb5877fe640 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x558f919cbfb0 0x558f919ce3d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd0916848b29c304 server_cookie=3dd5152c696ea716 in_seq=0 out_seq=0 2024-03-26T06:39:34.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:34.290+0000 7fb566ffd640 10 client.25066 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:34.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:33 smithi038 ceph-mon[32265]: pgmap v64: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 505 MiB/s wr, 9.24k op/s 2024-03-26T06:39:34.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:33 smithi121 ceph-mon[36349]: pgmap v64: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 505 MiB/s wr, 9.24k op/s 2024-03-26T06:39:34.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:33 smithi170 ceph-mon[36912]: pgmap v64: 97 pgs: 97 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 505 MiB/s wr, 9.24k op/s 2024-03-26T06:39:36.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:35 smithi038 ceph-mon[32265]: pgmap v65: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 617 MiB/s wr, 23.40k op/s 2024-03-26T06:39:36.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:35 smithi121 ceph-mon[36349]: pgmap v65: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 617 MiB/s wr, 23.40k op/s 2024-03-26T06:39:36.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:35 smithi170 ceph-mon[36912]: pgmap v65: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 617 MiB/s wr, 23.40k op/s 2024-03-26T06:39:38.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:37 smithi038 ceph-mon[32265]: pgmap v66: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 276 MiB/s wr, 23.27k op/s 2024-03-26T06:39:38.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:37 smithi121 ceph-mon[36349]: pgmap v66: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 276 MiB/s wr, 23.27k op/s 2024-03-26T06:39:38.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:37 smithi170 ceph-mon[36912]: pgmap v66: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 276 MiB/s wr, 23.27k op/s 2024-03-26T06:39:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.675+0000 7fb566ffd640 1 -- 172.21.15.38:0/3891901569 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7fb578023d80 con 0x558f919cbfb0 2024-03-26T06:39:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.675+0000 7fb566ffd640 10 client.25066 handle_command_reply: tid=0 2024-03-26T06:39:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.675+0000 7fb590677740 1 client.25066 shutdown 2024-03-26T06:39:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.676+0000 7fb590677740 1 -- 172.21.15.38:0/3891901569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f919617a0 msgr2=0x558f9196bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.676+0000 7fb590677740 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f919617a0 0x558f9196bc90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb57c09b2b0 tx=0x7fb57c097ba0 comp rx=0 tx=0).stop 2024-03-26T06:39:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.676+0000 7fb590677740 1 -- 172.21.15.38:0/3891901569 shutdown_connections 2024-03-26T06:39:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.676+0000 7fb590677740 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x558f919cbfb0 0x558f919ce3d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.676+0000 7fb590677740 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558f91964550 0x558f9195d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.676+0000 7fb590677740 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x558f919617a0 0x558f9196bc90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.676+0000 7fb590677740 1 --2- 172.21.15.38:0/3891901569 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x558f9195bd40 0x558f9196b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.676+0000 7fb590677740 1 -- 172.21.15.38:0/3891901569 >> 172.21.15.38:0/3891901569 conn(0x558f919225b0 msgr2=0x558f919609c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.676+0000 7fb566ffd640 0 client.25066 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.676+0000 7fb590677740 1 -- 172.21.15.38:0/3891901569 shutdown_connections 2024-03-26T06:39:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.676+0000 7fb590677740 1 -- 172.21.15.38:0/3891901569 wait complete. 2024-03-26T06:39:38.680 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.677+0000 7fb590677740 20 client.25066 trim_cache size 0 max 16384 2024-03-26T06:39:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:39:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:39:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:39:38.681 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:39:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.678+0000 7fb58da97640 1 -- 172.21.15.38:0/4230339581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb5700a2b60 msgr2=0x7fb5700a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.678+0000 7fb58da97640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb5700a2b60 0x7fb5700a5020 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb580032520 tx=0x7fb58002c140 comp rx=0 tx=0).stop 2024-03-26T06:39:38.682 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.678+0000 7fb58da97640 1 -- 172.21.15.38:0/4230339581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb58810fea0 msgr2=0x7fb5881acb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.678+0000 7fb58da97640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb58810fea0 0x7fb5881acb50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb57c00d770 tx=0x7fb57c00dc30 comp rx=0 tx=0).stop 2024-03-26T06:39:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.678+0000 7fb58da97640 1 -- 172.21.15.38:0/4230339581 shutdown_connections 2024-03-26T06:39:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.678+0000 7fb58da97640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb5700a2b60 0x7fb5700a5020 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.678+0000 7fb58da97640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb58811aad0 0x7fb5881a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.678+0000 7fb58da97640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb58810fea0 0x7fb5881acb50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.678+0000 7fb58da97640 1 --2- 172.21.15.38:0/4230339581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb58810fa30 0x7fb5881ac610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.679+0000 7fb58da97640 1 -- 172.21.15.38:0/4230339581 >> 172.21.15.38:0/4230339581 conn(0x7fb58806c390 msgr2=0x7fb588071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.679+0000 7fb58da97640 1 -- 172.21.15.38:0/4230339581 shutdown_connections 2024-03-26T06:39:38.683 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:38.679+0000 7fb58da97640 1 -- 172.21.15.38:0/4230339581 wait complete. 2024-03-26T06:39:39.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:38 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:39:39.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:38 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:39:39.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:38 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:39:40.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:40 smithi121 ceph-mon[36349]: pgmap v67: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 276 MiB/s wr, 23.27k op/s 2024-03-26T06:39:40.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:40 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:40.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:40 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:40.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:40 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:40.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:40 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:40.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:40 smithi170 ceph-mon[36912]: pgmap v67: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 276 MiB/s wr, 23.27k op/s 2024-03-26T06:39:40.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:40 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:40.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:40 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:40.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:40 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:40.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:40 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:40.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:40 smithi038 ceph-mon[32265]: pgmap v67: 97 pgs: 97 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 276 MiB/s wr, 23.27k op/s 2024-03-26T06:39:40.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:40 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:40.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:40 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:40.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:40 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:40.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:40 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:41.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:41 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:39:41.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:41 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:39:41.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:41 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:41.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:41 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:39:41.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:41 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:39:41.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:41 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:41.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:41 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:39:41.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:41 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:39:41.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:41 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:39:42.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:42 smithi121 ceph-mon[36349]: pgmap v68: 97 pgs: 97 active+clean; 13 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 412 MiB/s wr, 36.74k op/s 2024-03-26T06:39:42.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:42 smithi170 ceph-mon[36912]: pgmap v68: 97 pgs: 97 active+clean; 13 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 412 MiB/s wr, 36.74k op/s 2024-03-26T06:39:42.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:42 smithi038 ceph-mon[32265]: pgmap v68: 97 pgs: 97 active+clean; 13 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 412 MiB/s wr, 36.74k op/s 2024-03-26T06:39:44.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:44 smithi121 ceph-mon[36349]: pgmap v69: 97 pgs: 97 active+clean; 13 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 247 MiB/s wr, 27.64k op/s 2024-03-26T06:39:44.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:44 smithi170 ceph-mon[36912]: pgmap v69: 97 pgs: 97 active+clean; 13 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 247 MiB/s wr, 27.64k op/s 2024-03-26T06:39:44.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:44 smithi038 ceph-mon[32265]: pgmap v69: 97 pgs: 97 active+clean; 13 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 247 MiB/s wr, 27.64k op/s 2024-03-26T06:39:46.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:46 smithi121 ceph-mon[36349]: pgmap v70: 97 pgs: 97 active+clean; 13 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 392 MiB/s wr, 39.58k op/s 2024-03-26T06:39:46.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:46 smithi170 ceph-mon[36912]: pgmap v70: 97 pgs: 97 active+clean; 13 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 392 MiB/s wr, 39.58k op/s 2024-03-26T06:39:46.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:46 smithi038 ceph-mon[32265]: pgmap v70: 97 pgs: 97 active+clean; 13 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 392 MiB/s wr, 39.58k op/s 2024-03-26T06:39:48.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:48 smithi121 ceph-mon[36349]: pgmap v71: 97 pgs: 97 active+clean; 13 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 280 MiB/s wr, 25.41k op/s 2024-03-26T06:39:48.417 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:48 smithi170 ceph-mon[36912]: pgmap v71: 97 pgs: 97 active+clean; 13 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 280 MiB/s wr, 25.41k op/s 2024-03-26T06:39:48.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:48 smithi038 ceph-mon[32265]: pgmap v71: 97 pgs: 97 active+clean; 13 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 280 MiB/s wr, 25.41k op/s 2024-03-26T06:39:50.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:50 smithi121 ceph-mon[36349]: pgmap v72: 97 pgs: 97 active+clean; 13 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 280 MiB/s wr, 25.41k op/s 2024-03-26T06:39:50.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:50 smithi170 ceph-mon[36912]: pgmap v72: 97 pgs: 97 active+clean; 13 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 280 MiB/s wr, 25.41k op/s 2024-03-26T06:39:50.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:50 smithi038 ceph-mon[32265]: pgmap v72: 97 pgs: 97 active+clean; 13 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 280 MiB/s wr, 25.41k op/s 2024-03-26T06:39:51.861 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.858+0000 7f29c7580640 1 -- 172.21.15.38:0/2668809644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29c010b8a0 msgr2=0x7f29c010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:51.862 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.858+0000 7f29c7580640 1 --2- 172.21.15.38:0/2668809644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29c010b8a0 0x7f29c010dd40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009900 tx=0x7f29b002bd70 comp rx=0 tx=0).stop 2024-03-26T06:39:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.861+0000 7f29c7580640 1 -- 172.21.15.38:0/2668809644 shutdown_connections 2024-03-26T06:39:51.863 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.861+0000 7f29c7580640 1 --2- 172.21.15.38:0/2668809644 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29c010b8a0 0x7f29c010dd40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.861+0000 7f29c7580640 1 --2- 172.21.15.38:0/2668809644 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29c0102e80 0x7f29c010b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.861+0000 7f29c7580640 1 --2- 172.21.15.38:0/2668809644 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29c0102560 0x7f29c0102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.861+0000 7f29c7580640 1 -- 172.21.15.38:0/2668809644 >> 172.21.15.38:0/2668809644 conn(0x7f29c0077320 msgr2=0x7f29c0077730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:51.864 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.861+0000 7f29c7580640 1 -- 172.21.15.38:0/2668809644 shutdown_connections 2024-03-26T06:39:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.861+0000 7f29c7580640 1 -- 172.21.15.38:0/2668809644 wait complete. 2024-03-26T06:39:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.862+0000 7f29c7580640 1 Processor -- start 2024-03-26T06:39:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.862+0000 7f29c7580640 1 -- start start 2024-03-26T06:39:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.863+0000 7f29c7580640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29c0102560 0x7f29c010a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.863+0000 7f29c7580640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29c0102e80 0x7f29c010aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.863+0000 7f29c7580640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29c010b8a0 0x7f29c01b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.863+0000 7f29c7580640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c0111730 con 0x7f29c0102e80 2024-03-26T06:39:51.865 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.863+0000 7f29c7580640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c01115b0 con 0x7f29c010b8a0 2024-03-26T06:39:51.866 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.863+0000 7f29c7580640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c01118b0 con 0x7f29c0102560 2024-03-26T06:39:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.864+0000 7f29c531c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29c0102560 0x7f29c010a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.864+0000 7f29c531c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29c0102560 0x7f29c010a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:43596/0 (socket says 172.21.15.38:43596) 2024-03-26T06:39:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.864+0000 7f29c531c640 1 -- 172.21.15.38:0/1900871889 learned_addr learned my addr 172.21.15.38:0/1900871889 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.864+0000 7f29c5b1d640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29c010b8a0 0x7f29c01b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.867 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.864+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29c0102e80 0x7f29c010aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.865+0000 7f29c4b1b640 1 -- 172.21.15.38:0/1900871889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29c0102560 msgr2=0x7f29c010a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.865+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29c0102560 0x7f29c010a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.865+0000 7f29c4b1b640 1 -- 172.21.15.38:0/1900871889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29c010b8a0 msgr2=0x7f29c01b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.865+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29c010b8a0 0x7f29c01b2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.868 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.865+0000 7f29c4b1b640 1 -- 172.21.15.38:0/1900871889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c006ae10 con 0x7f29c0102e80 2024-03-26T06:39:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.866+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29c0102e80 0x7f29c010aa50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00cce0 tx=0x7f29bc0075f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.866+0000 7f29b67fc640 1 -- 172.21.15.38:0/1900871889 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc004b20 con 0x7f29c0102e80 2024-03-26T06:39:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.866+0000 7f29c7580640 1 -- 172.21.15.38:0/1900871889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c006b100 con 0x7f29c0102e80 2024-03-26T06:39:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.866+0000 7f29b67fc640 1 -- 172.21.15.38:0/1900871889 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29bc004cc0 con 0x7f29c0102e80 2024-03-26T06:39:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.866+0000 7f29b67fc640 1 -- 172.21.15.38:0/1900871889 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc00f6f0 con 0x7f29c0102e80 2024-03-26T06:39:51.869 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.867+0000 7f29c7580640 1 -- 172.21.15.38:0/1900871889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c006b650 con 0x7f29c0102e80 2024-03-26T06:39:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.872+0000 7f29ca235740 1 Processor -- start 2024-03-26T06:39:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.872+0000 7f29ca235740 1 -- start start 2024-03-26T06:39:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.872+0000 7f29ca235740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d876d40 0x56148d875a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.872+0000 7f29ca235740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 0x56148d876450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.872+0000 7f29ca235740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d87f550 0x56148d881940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.872+0000 7f29ca235740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x56148d7da5f0 con 0x56148d876d40 2024-03-26T06:39:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.872+0000 7f29ca235740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56148d783930 con 0x56148d875fd0 2024-03-26T06:39:51.875 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.872+0000 7f29ca235740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56148d7d6690 con 0x56148d87f550 2024-03-26T06:39:51.876 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.872+0000 7f29c5b1d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d876d40 0x56148d875a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.873+0000 7f29b67fc640 1 -- 172.21.15.38:0/1900871889 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f29bc00f8d0 con 0x7f29c0102e80 2024-03-26T06:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.874+0000 7f29b67fc640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f29ac0a2bb0 0x7f29ac0a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.877 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.874+0000 7f29c531c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 0x56148d876450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.874+0000 7f29c531c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 0x56148d876450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:53952/0 (socket says 172.21.15.38:53952) 2024-03-26T06:39:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.874+0000 7f29c531c640 1 -- 172.21.15.38:0/2991029701 learned_addr learned my addr 172.21.15.38:0/2991029701 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.875+0000 7f29c531c640 1 -- 172.21.15.38:0/2991029701 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d87f550 msgr2=0x56148d881940 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-26T06:39:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.875+0000 7f29c531c640 1 --2- 172.21.15.38:0/2991029701 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d87f550 0x56148d881940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.875+0000 7f29c531c640 1 -- 172.21.15.38:0/2991029701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d876d40 msgr2=0x56148d875a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:51.878 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.875+0000 7f29c531c640 1 --2- 172.21.15.38:0/2991029701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d876d40 0x56148d875a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.875+0000 7f29c531c640 1 -- 172.21.15.38:0/2991029701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56148d4ef210 con 0x56148d875fd0 2024-03-26T06:39:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.876+0000 7f29b67fc640 1 -- 172.21.15.38:0/1900871889 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f29bc011d30 con 0x7f29c0102e80 2024-03-26T06:39:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.876+0000 7f29c5b1d640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f29ac0a2bb0 0x7f29ac0a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.876+0000 7f29c531c640 1 --2- 172.21.15.38:0/2991029701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 0x56148d876450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f29b8007890 tx=0x7f29b8007d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd78d0e697d66de9 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:51.879 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.877+0000 7f29ab7fe640 1 -- 172.21.15.38:0/2991029701 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b800d950 con 0x56148d875fd0 2024-03-26T06:39:51.880 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.877+0000 7f29c5b1d640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f29ac0a2bb0 0x7f29ac0a5070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f29b0005ab0 tx=0x7f29b0009260 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:51.882 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.880+0000 7f29ab7fe640 1 -- 172.21.15.38:0/2991029701 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29b8018070 con 0x56148d875fd0 2024-03-26T06:39:51.883 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.881+0000 7f29ca235740 1 -- 172.21.15.38:0/2991029701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 msgr2=0x56148d876450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:51.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.886+0000 7f29ab7fe640 1 -- 172.21.15.38:0/2991029701 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b80123e0 con 0x56148d875fd0 2024-03-26T06:39:51.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.886+0000 7f29ca235740 1 --2- 172.21.15.38:0/2991029701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 0x56148d876450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f29b8007890 tx=0x7f29b8007d50 comp rx=0 tx=0).stop 2024-03-26T06:39:51.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.887+0000 7f29ca235740 1 -- 172.21.15.38:0/2991029701 shutdown_connections 2024-03-26T06:39:51.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.887+0000 7f29ca235740 1 --2- 172.21.15.38:0/2991029701 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d87f550 0x56148d881940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.889 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.887+0000 7f29ca235740 1 --2- 172.21.15.38:0/2991029701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 0x56148d876450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.887+0000 7f29ca235740 1 --2- 172.21.15.38:0/2991029701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d876d40 0x56148d875a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.890 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.887+0000 7f29ca235740 1 -- 172.21.15.38:0/2991029701 >> 172.21.15.38:0/2991029701 conn(0x56148d83d5b0 msgr2=0x56148d870cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:51.895 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.893+0000 7f29ca235740 1 -- 172.21.15.38:0/2991029701 shutdown_connections 2024-03-26T06:39:51.897 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.894+0000 7f29ca235740 1 -- 172.21.15.38:0/2991029701 wait complete. 2024-03-26T06:39:51.897 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.894+0000 7f29ca235740 1 -- start start 2024-03-26T06:39:51.897 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.895+0000 7f29ca235740 1 Processor -- start 2024-03-26T06:39:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.895+0000 7f29ca235740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d875fd0 0x56148d87c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.895+0000 7f29ca235740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 0x56148d87ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.895+0000 7f29ca235740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d87f550 0x56148d87cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.895+0000 7f29ca235740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x56148d8822d0 con 0x56148d87f550 2024-03-26T06:39:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.895+0000 7f29ca235740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56148d796d90 con 0x56148d876d40 2024-03-26T06:39:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.895+0000 7f29ca235740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56148d882450 con 0x56148d875fd0 2024-03-26T06:39:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.895+0000 7f29ca235740 10 client.0 _ll_register_callbacks cb 0x56148d4eea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:39:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.896+0000 7f29c4b1b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 0x56148d87ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.896+0000 7f29c4b1b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 0x56148d87ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:53956/0 (socket says 172.21.15.38:53956) 2024-03-26T06:39:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.896+0000 7f29c4b1b640 1 -- 172.21.15.38:0/3976613654 learned_addr learned my addr 172.21.15.38:0/3976613654 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:51.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.896+0000 7f29c531c640 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d875fd0 0x56148d87c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.898+0000 7f29c5b1d640 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d87f550 0x56148d87cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.898+0000 7f29c531c640 1 -- 172.21.15.38:0/3976613654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 msgr2=0x56148d87ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.898+0000 7f29c531c640 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 0x56148d87ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.901 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.898+0000 7f29c531c640 1 -- 172.21.15.38:0/3976613654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d87f550 msgr2=0x56148d87cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.898+0000 7f29c531c640 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d87f550 0x56148d87cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.898+0000 7f29c531c640 1 -- 172.21.15.38:0/3976613654 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56148d5bb620 con 0x56148d875fd0 2024-03-26T06:39:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.899+0000 7f29c531c640 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d875fd0 0x56148d87c500 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f29b8000c00 tx=0x7f29b80137a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3579e588d5eca6a server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:51.902 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.899+0000 7f29a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-26T06:39:51.903 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.901+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3976613654 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b800dd60 con 0x56148d875fd0 2024-03-26T06:39:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.901+0000 7f29ca235740 1 -- 172.21.15.38:0/3976613654 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f29b8018070 con 0x56148d875fd0 2024-03-26T06:39:51.904 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.901+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3976613654 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f29b801c3f0 con 0x56148d875fd0 2024-03-26T06:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.906+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3976613654 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29b801c610 con 0x56148d875fd0 2024-03-26T06:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.906+0000 7f29ca235740 10 client.25072 fetch_fsmap learned FSMap version 27 2024-03-26T06:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.906+0000 7f29ca235740 1 -- 172.21.15.38:0/3976613654 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56148d88e580 con 0x56148d875fd0 2024-03-26T06:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.907+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3976613654 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29b8018070 con 0x56148d875fd0 2024-03-26T06:39:51.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.907+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3976613654 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f29b802c7f0 con 0x56148d875fd0 2024-03-26T06:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.907+0000 7f29ca235740 10 client.25072 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.907+0000 7f29ca235740 10 client.25072 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.907+0000 7f29ca235740 20 client.25072 populate_metadata read hostname 'smithi038' 2024-03-26T06:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.907+0000 7f29ca235740 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56148d88fd70 0x56148d8921b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.907+0000 7f29ca235740 4 client.25072 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:39:51.910 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.907+0000 7f29ca235740 1 -- 172.21.15.38:0/3976613654 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56148d7e7b30 con 0x56148d88fd70 2024-03-26T06:39:51.911 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.908+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56148d88fd70 0x56148d8921b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.917+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56148d88fd70 0x56148d8921b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9f625d2f685580e server_cookie=76188f93b493fe2a in_seq=0 out_seq=0 2024-03-26T06:39:51.919 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.917+0000 7f29a8ff9640 10 client.25072 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.921+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3976613654 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56148d7e7b30 con 0x56148d88fd70 2024-03-26T06:39:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.921+0000 7f29a8ff9640 10 client.25072 handle_command_reply: tid=0 2024-03-26T06:39:51.924 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.921+0000 7f29ca235740 1 client.25072 shutdown 2024-03-26T06:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.922+0000 7f29ca235740 1 -- 172.21.15.38:0/3976613654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d875fd0 msgr2=0x56148d87c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:51.925 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.922+0000 7f29ca235740 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d875fd0 0x56148d87c500 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f29b8000c00 tx=0x7f29b80137a0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.935 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.933+0000 7f29ca235740 1 -- 172.21.15.38:0/3976613654 shutdown_connections 2024-03-26T06:39:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.933+0000 7f29ca235740 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56148d88fd70 0x56148d8921b0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.933+0000 7f29ca235740 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d87f550 0x56148d87cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.933+0000 7f29ca235740 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 0x56148d87ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.933+0000 7f29ca235740 1 --2- 172.21.15.38:0/3976613654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d875fd0 0x56148d87c500 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.933+0000 7f29ca235740 1 -- 172.21.15.38:0/3976613654 >> 172.21.15.38:0/3976613654 conn(0x56148d83d5b0 msgr2=0x56148d872a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:51.936 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.933+0000 7f29a8ff9640 0 client.25072 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.933+0000 7f29ca235740 1 -- 172.21.15.38:0/3976613654 shutdown_connections 2024-03-26T06:39:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.933+0000 7f29ca235740 1 -- 172.21.15.38:0/3976613654 wait complete. 2024-03-26T06:39:51.937 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.933+0000 7f29ca235740 20 client.25072 trim_cache size 0 max 16384 2024-03-26T06:39:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.978+0000 7f29ca235740 1 Processor -- start 2024-03-26T06:39:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.982+0000 7f29ca235740 1 -- start start 2024-03-26T06:39:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.982+0000 7f29ca235740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 0x56148d87ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.982+0000 7f29ca235740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d876d40 0x56148d87cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.982+0000 7f29ca235740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d87f550 0x56148d896b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.983+0000 7f29c531c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 0x56148d87ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.983+0000 7f29c531c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 0x56148d87ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:53968/0 (socket says 172.21.15.38:53968) 2024-03-26T06:39:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.983+0000 7f29c4b1b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d876d40 0x56148d87cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.983+0000 7f29c4b1b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d876d40 0x56148d87cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:43610/0 (socket says 172.21.15.38:43610) 2024-03-26T06:39:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.983+0000 7f29c531c640 1 -- 172.21.15.38:0/2118709801 learned_addr learned my addr 172.21.15.38:0/2118709801 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:51.986 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.983+0000 7f29ca235740 1 -- 172.21.15.38:0/2118709801 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c01c6ae0 con 0x56148d87f550 2024-03-26T06:39:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.983+0000 7f29ca235740 1 -- 172.21.15.38:0/2118709801 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c01c4620 con 0x56148d875fd0 2024-03-26T06:39:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.983+0000 7f29ca235740 1 -- 172.21.15.38:0/2118709801 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c0196c00 con 0x56148d876d40 2024-03-26T06:39:51.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.984+0000 7f29c4b1b640 1 -- 172.21.15.38:0/2118709801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 msgr2=0x56148d87ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.984+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/2118709801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 0x56148d87ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.984+0000 7f29c4b1b640 1 -- 172.21.15.38:0/2118709801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d87f550 msgr2=0x56148d896b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.984+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/2118709801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d87f550 0x56148d896b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.984+0000 7f29c4b1b640 1 -- 172.21.15.38:0/2118709801 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56148d8752a0 con 0x56148d876d40 2024-03-26T06:39:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.984+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/2118709801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d876d40 0x56148d87cda0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f29bc09a850 tx=0x7f29bc007f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27bd892079f5f0f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.985+0000 7f29aaffd640 1 -- 172.21.15.38:0/2118709801 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc0b8070 con 0x56148d876d40 2024-03-26T06:39:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.985+0000 7f29aaffd640 1 -- 172.21.15.38:0/2118709801 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29bc09a8c0 con 0x56148d876d40 2024-03-26T06:39:51.988 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.986+0000 7f29ca235740 1 -- 172.21.15.38:0/2118709801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d876d40 msgr2=0x56148d87cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:51.989 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.986+0000 7f29ca235740 1 --2- 172.21.15.38:0/2118709801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d876d40 0x56148d87cda0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f29bc09a850 tx=0x7f29bc007f00 comp rx=0 tx=0).stop 2024-03-26T06:39:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.992+0000 7f29ca235740 1 -- 172.21.15.38:0/2118709801 shutdown_connections 2024-03-26T06:39:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.992+0000 7f29ca235740 1 --2- 172.21.15.38:0/2118709801 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d87f550 0x56148d896b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.992+0000 7f29ca235740 1 --2- 172.21.15.38:0/2118709801 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d876d40 0x56148d87cda0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.992+0000 7f29ca235740 1 --2- 172.21.15.38:0/2118709801 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d875fd0 0x56148d87ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:51.998 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.992+0000 7f29ca235740 1 -- 172.21.15.38:0/2118709801 >> 172.21.15.38:0/2118709801 conn(0x56148d83d5b0 msgr2=0x56148d8778b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.992+0000 7f29ca235740 1 -- 172.21.15.38:0/2118709801 shutdown_connections 2024-03-26T06:39:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.992+0000 7f29ca235740 1 -- 172.21.15.38:0/2118709801 wait complete. 2024-03-26T06:39:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.993+0000 7f29ca235740 1 -- start start 2024-03-26T06:39:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.993+0000 7f29ca235740 1 Processor -- start 2024-03-26T06:39:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.994+0000 7f29ca235740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d875fd0 0x56148d884700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:51.999 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.994+0000 7f29ca235740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 0x56148d884c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.994+0000 7f29ca235740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d87f550 0x56148d885200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.994+0000 7f29ca235740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c0072860 con 0x56148d875fd0 2024-03-26T06:39:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.994+0000 7f29c4b1b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 0x56148d884c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.995+0000 7f29c4b1b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 0x56148d884c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:53972/0 (socket says 172.21.15.38:53972) 2024-03-26T06:39:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.995+0000 7f29c4b1b640 1 -- 172.21.15.38:0/3196099702 learned_addr learned my addr 172.21.15.38:0/3196099702 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:39:52.000 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.995+0000 7f29ca235740 1 -- 172.21.15.38:0/3196099702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c01c3fc0 con 0x56148d876d40 2024-03-26T06:39:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.995+0000 7f29ca235740 1 -- 172.21.15.38:0/3196099702 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c0072e60 con 0x56148d87f550 2024-03-26T06:39:52.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.995+0000 7f29ca235740 10 client.0 _ll_register_callbacks cb 0x7f29b80072c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:39:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.995+0000 7f29c531c640 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d875fd0 0x56148d884700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.996+0000 7f29c5b1d640 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d87f550 0x56148d885200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:52.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.996+0000 7f29c4b1b640 1 -- 172.21.15.38:0/3196099702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d87f550 msgr2=0x56148d885200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:52.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.996+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d87f550 0x56148d885200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:52.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.996+0000 7f29c4b1b640 1 -- 172.21.15.38:0/3196099702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d875fd0 msgr2=0x56148d884700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:39:52.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.996+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d875fd0 0x56148d884700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:52.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.996+0000 7f29c4b1b640 1 -- 172.21.15.38:0/3196099702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56148d8e55a0 con 0x56148d876d40 2024-03-26T06:39:52.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.997+0000 7f29c4b1b640 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 0x56148d884c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f29bc09a5c0 tx=0x7f29bc0cbe60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c563ff5d3abcb04 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:39:52.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.997+0000 7f29a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-03-26T06:39:52.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.997+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3196099702 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29bc0b8070 con 0x56148d876d40 2024-03-26T06:39:52.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.997+0000 7f29ca235740 1 -- 172.21.15.38:0/3196099702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f29bc09a8c0 con 0x56148d876d40 2024-03-26T06:39:52.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.998+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3196099702 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f29bc096690 con 0x56148d876d40 2024-03-26T06:39:52.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.998+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3196099702 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29bc09ea10 con 0x56148d876d40 2024-03-26T06:39:52.007 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.998+0000 7f29ca235740 10 client.25187 fetch_fsmap learned FSMap version 27 2024-03-26T06:39:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.998+0000 7f29ca235740 1 -- 172.21.15.38:0/3196099702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56148d8e68b0 con 0x56148d876d40 2024-03-26T06:39:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:51.999+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3196099702 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29bc0a3060 con 0x56148d876d40 2024-03-26T06:39:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:52.000+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3196099702 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f29bc0ba430 con 0x56148d876d40 2024-03-26T06:39:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:52.000+0000 7f29ca235740 10 client.25187 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:39:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:52.000+0000 7f29ca235740 10 client.25187 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:39:52.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:52.000+0000 7f29ca235740 20 client.25187 populate_metadata read hostname 'smithi038' 2024-03-26T06:39:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:52.000+0000 7f29ca235740 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56148d8e6f00 0x56148d8e9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:39:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:52.000+0000 7f29ca235740 4 client.25187 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:39:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:52.001+0000 7f29ca235740 1 -- 172.21.15.38:0/3196099702 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7f29bc09a8c0 con 0x56148d8e6f00 2024-03-26T06:39:52.009 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:52.002+0000 7f29c531c640 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56148d8e6f00 0x56148d8e9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:39:52.015 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:52.013+0000 7f29c531c640 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56148d8e6f00 0x56148d8e9320 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee1d1522363c4835 server_cookie=bbd6eae5c656665a in_seq=0 out_seq=0 2024-03-26T06:39:52.016 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:52.013+0000 7f29a8ff9640 10 client.25187 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:52.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:52 smithi170 ceph-mon[36912]: pgmap v73: 97 pgs: 97 active+clean; 13 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 453 MiB/s wr, 37.37k op/s 2024-03-26T06:39:52.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:52 smithi038 ceph-mon[32265]: pgmap v73: 97 pgs: 97 active+clean; 13 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 453 MiB/s wr, 37.37k op/s 2024-03-26T06:39:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:52 smithi121 ceph-mon[36349]: pgmap v73: 97 pgs: 97 active+clean; 13 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 453 MiB/s wr, 37.37k op/s 2024-03-26T06:39:54.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:54 smithi170 ceph-mon[36912]: pgmap v74: 97 pgs: 97 active+clean; 13 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 581 KiB/s rd, 318 MiB/s wr, 23.90k op/s 2024-03-26T06:39:54.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:54 smithi038 ceph-mon[32265]: pgmap v74: 97 pgs: 97 active+clean; 13 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 581 KiB/s rd, 318 MiB/s wr, 23.90k op/s 2024-03-26T06:39:54.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:54 smithi121 ceph-mon[36349]: pgmap v74: 97 pgs: 97 active+clean; 13 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 581 KiB/s rd, 318 MiB/s wr, 23.90k op/s 2024-03-26T06:39:56.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:56 smithi170 ceph-mon[36912]: pgmap v75: 97 pgs: 97 active+clean; 13 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 431 MiB/s wr, 33.23k op/s 2024-03-26T06:39:56.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:56 smithi038 ceph-mon[32265]: pgmap v75: 97 pgs: 97 active+clean; 13 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 431 MiB/s wr, 33.23k op/s 2024-03-26T06:39:56.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:56 smithi121 ceph-mon[36349]: pgmap v75: 97 pgs: 97 active+clean; 13 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 1.0 MiB/s rd, 431 MiB/s wr, 33.23k op/s 2024-03-26T06:39:58.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:39:58 smithi038 ceph-mon[32265]: pgmap v76: 97 pgs: 97 active+clean; 13 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 728 KiB/s rd, 318 MiB/s wr, 23.94k op/s 2024-03-26T06:39:58.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:39:58 smithi121 ceph-mon[36349]: pgmap v76: 97 pgs: 97 active+clean; 13 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 728 KiB/s rd, 318 MiB/s wr, 23.94k op/s 2024-03-26T06:39:58.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:39:58 smithi170 ceph-mon[36912]: pgmap v76: 97 pgs: 97 active+clean; 13 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 728 KiB/s rd, 318 MiB/s wr, 23.94k op/s 2024-03-26T06:39:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29a8ff9640 1 -- 172.21.15.38:0/3196099702 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f29bc09a8c0 con 0x56148d8e6f00 2024-03-26T06:39:58.764 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29a8ff9640 10 client.25187 handle_command_reply: tid=0 2024-03-26T06:39:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29ca235740 1 client.25187 shutdown 2024-03-26T06:39:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29ca235740 1 -- 172.21.15.38:0/3196099702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 msgr2=0x56148d884c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:58.765 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29ca235740 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 0x56148d884c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f29bc09a5c0 tx=0x7f29bc0cbe60 comp rx=0 tx=0).stop 2024-03-26T06:39:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29ca235740 1 -- 172.21.15.38:0/3196099702 shutdown_connections 2024-03-26T06:39:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29ca235740 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56148d8e6f00 0x56148d8e9320 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29ca235740 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56148d87f550 0x56148d885200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29ca235740 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56148d876d40 0x56148d884c40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29ca235740 1 --2- 172.21.15.38:0/3196099702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56148d875fd0 0x56148d884700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29ca235740 1 -- 172.21.15.38:0/3196099702 >> 172.21.15.38:0/3196099702 conn(0x56148d83d5b0 msgr2=0x56148d896b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.762+0000 7f29a8ff9640 0 client.25187 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:39:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.763+0000 7f29ca235740 1 -- 172.21.15.38:0/3196099702 shutdown_connections 2024-03-26T06:39:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.763+0000 7f29ca235740 1 -- 172.21.15.38:0/3196099702 wait complete. 2024-03-26T06:39:58.766 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.763+0000 7f29ca235740 20 client.25187 trim_cache size 0 max 16384 2024-03-26T06:39:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:39:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:39:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:39:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:39:58.767 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.763+0000 7f29c7580640 1 -- 172.21.15.38:0/1900871889 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f29ac0a2bb0 msgr2=0x7f29ac0a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.763+0000 7f29c7580640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f29ac0a2bb0 0x7f29ac0a5070 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f29b0005ab0 tx=0x7f29b0009260 comp rx=0 tx=0).stop 2024-03-26T06:39:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.764+0000 7f29c7580640 1 -- 172.21.15.38:0/1900871889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29c0102e80 msgr2=0x7f29c010aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:39:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.764+0000 7f29c7580640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29c0102e80 0x7f29c010aa50 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f29bc00cce0 tx=0x7f29bc0075f0 comp rx=0 tx=0).stop 2024-03-26T06:39:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.764+0000 7f29c7580640 1 -- 172.21.15.38:0/1900871889 shutdown_connections 2024-03-26T06:39:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.764+0000 7f29c7580640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f29ac0a2bb0 0x7f29ac0a5070 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.764+0000 7f29c7580640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29c010b8a0 0x7f29c01b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.764+0000 7f29c7580640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29c0102e80 0x7f29c010aa50 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.764+0000 7f29c7580640 1 --2- 172.21.15.38:0/1900871889 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f29c0102560 0x7f29c010a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:39:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.764+0000 7f29c7580640 1 -- 172.21.15.38:0/1900871889 >> 172.21.15.38:0/1900871889 conn(0x7f29c0077320 msgr2=0x7f29c0103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:39:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.764+0000 7f29c7580640 1 -- 172.21.15.38:0/1900871889 shutdown_connections 2024-03-26T06:39:58.768 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:39:58.764+0000 7f29c7580640 1 -- 172.21.15.38:0/1900871889 wait complete. 2024-03-26T06:40:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:00 smithi038 ceph-mon[32265]: pgmap v77: 97 pgs: 97 active+clean; 13 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 728 KiB/s rd, 318 MiB/s wr, 23.94k op/s 2024-03-26T06:40:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:00 smithi038 ceph-mon[32265]: overall HEALTH_OK 2024-03-26T06:40:00.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:00 smithi121 ceph-mon[36349]: pgmap v77: 97 pgs: 97 active+clean; 13 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 728 KiB/s rd, 318 MiB/s wr, 23.94k op/s 2024-03-26T06:40:00.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:00 smithi121 ceph-mon[36349]: overall HEALTH_OK 2024-03-26T06:40:00.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:00 smithi170 ceph-mon[36912]: pgmap v77: 97 pgs: 97 active+clean; 13 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 728 KiB/s rd, 318 MiB/s wr, 23.94k op/s 2024-03-26T06:40:00.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:00 smithi170 ceph-mon[36912]: overall HEALTH_OK 2024-03-26T06:40:02.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:02 smithi038 ceph-mon[32265]: pgmap v78: 97 pgs: 97 active+clean; 13 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 447 MiB/s wr, 35.49k op/s 2024-03-26T06:40:02.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:02 smithi121 ceph-mon[36349]: pgmap v78: 97 pgs: 97 active+clean; 13 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 447 MiB/s wr, 35.49k op/s 2024-03-26T06:40:02.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:02 smithi170 ceph-mon[36912]: pgmap v78: 97 pgs: 97 active+clean; 13 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 1.1 MiB/s rd, 447 MiB/s wr, 35.49k op/s 2024-03-26T06:40:04.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:04 smithi038 ceph-mon[32265]: pgmap v79: 97 pgs: 97 active+clean; 13 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 849 KiB/s rd, 273 MiB/s wr, 23.53k op/s 2024-03-26T06:40:04.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:04 smithi121 ceph-mon[36349]: pgmap v79: 97 pgs: 97 active+clean; 13 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 849 KiB/s rd, 273 MiB/s wr, 23.53k op/s 2024-03-26T06:40:04.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:04 smithi170 ceph-mon[36912]: pgmap v79: 97 pgs: 97 active+clean; 13 GiB data, 57 GiB used, 1016 GiB / 1.0 TiB avail; 849 KiB/s rd, 273 MiB/s wr, 23.53k op/s 2024-03-26T06:40:06.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:06 smithi038 ceph-mon[32265]: pgmap v80: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 1006 KiB/s rd, 344 MiB/s wr, 30.29k op/s 2024-03-26T06:40:06.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:06 smithi121 ceph-mon[36349]: pgmap v80: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 1006 KiB/s rd, 344 MiB/s wr, 30.29k op/s 2024-03-26T06:40:06.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:06 smithi170 ceph-mon[36912]: pgmap v80: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 1006 KiB/s rd, 344 MiB/s wr, 30.29k op/s 2024-03-26T06:40:08.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:08 smithi038 ceph-mon[32265]: pgmap v81: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 553 KiB/s rd, 277 MiB/s wr, 25.46k op/s 2024-03-26T06:40:08.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:08 smithi121 ceph-mon[36349]: pgmap v81: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 553 KiB/s rd, 277 MiB/s wr, 25.46k op/s 2024-03-26T06:40:08.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:08 smithi170 ceph-mon[36912]: pgmap v81: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 553 KiB/s rd, 277 MiB/s wr, 25.46k op/s 2024-03-26T06:40:10.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:10 smithi038 ceph-mon[32265]: pgmap v82: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 553 KiB/s rd, 246 MiB/s wr, 22.82k op/s 2024-03-26T06:40:10.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:10 smithi121 ceph-mon[36349]: pgmap v82: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 553 KiB/s rd, 246 MiB/s wr, 22.82k op/s 2024-03-26T06:40:10.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:10 smithi170 ceph-mon[36912]: pgmap v82: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 553 KiB/s rd, 246 MiB/s wr, 22.82k op/s 2024-03-26T06:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.938+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3729194380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f1810ca40 msgr2=0x7f4f18079600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.938+0000 7f4f1d8fe640 1 --2- 172.21.15.38:0/3729194380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f1810ca40 0x7f4f18079600 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f4f10009be0 tx=0x7f4f1002bfc0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.938+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3729194380 shutdown_connections 2024-03-26T06:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.938+0000 7f4f1d8fe640 1 --2- 172.21.15.38:0/3729194380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f1810ca40 0x7f4f18079600 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.938+0000 7f4f1d8fe640 1 --2- 172.21.15.38:0/3729194380 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f1810df60 0x7f4f1810c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.941 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.938+0000 7f4f1d8fe640 1 --2- 172.21.15.38:0/3729194380 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f1810d640 0x7f4f1810da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.938+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3729194380 >> 172.21.15.38:0/3729194380 conn(0x7f4f1806c200 msgr2=0x7f4f1806c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:11.942 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.940+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3729194380 shutdown_connections 2024-03-26T06:40:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.940+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3729194380 wait complete. 2024-03-26T06:40:11.943 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.941+0000 7f4f1d8fe640 1 Processor -- start 2024-03-26T06:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.941+0000 7f4f1d8fe640 1 -- start start 2024-03-26T06:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.941+0000 7f4f1d8fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f1810ca40 0x7f4f18083fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.941+0000 7f4f1d8fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f1810d640 0x7f4f1807e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.941+0000 7f4f1d8fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f1810df60 0x7f4f1807e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.941+0000 7f4f1d8fe640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1807c6c0 con 0x7f4f1810df60 2024-03-26T06:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.941+0000 7f4f1d8fe640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1807c540 con 0x7f4f1810d640 2024-03-26T06:40:11.944 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.941+0000 7f4f1d8fe640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1807c840 con 0x7f4f1810ca40 2024-03-26T06:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.942+0000 7f4f1d0fd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f1810df60 0x7f4f1807e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.942+0000 7f4f1c8fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f1810ca40 0x7f4f18083fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.942+0000 7f4f1c8fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f1810ca40 0x7f4f18083fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:58412/0 (socket says 172.21.15.38:58412) 2024-03-26T06:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.942+0000 7f4f1c8fc640 1 -- 172.21.15.38:0/3747675926 learned_addr learned my addr 172.21.15.38:0/3747675926 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:11.945 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.942+0000 7f4f17fff640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f1810d640 0x7f4f1807e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.943+0000 7f4f1c8fc640 1 -- 172.21.15.38:0/3747675926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f1810d640 msgr2=0x7f4f1807e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.943+0000 7f4f1c8fc640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f1810d640 0x7f4f1807e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.943+0000 7f4f1c8fc640 1 -- 172.21.15.38:0/3747675926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f1810df60 msgr2=0x7f4f1807e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.943+0000 7f4f1c8fc640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f1810df60 0x7f4f1807e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.943+0000 7f4f1c8fc640 1 -- 172.21.15.38:0/3747675926 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f181365a0 con 0x7f4f1810ca40 2024-03-26T06:40:11.946 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.943+0000 7f4f1c8fc640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f1810ca40 0x7f4f18083fb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4f0800d950 tx=0x7f4f0800dd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:11.947 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.944+0000 7f4f15ffb640 1 -- 172.21.15.38:0/3747675926 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08017070 con 0x7f4f1810ca40 2024-03-26T06:40:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.944+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3747675926 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f18136890 con 0x7f4f1810ca40 2024-03-26T06:40:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.944+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3747675926 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f18136e00 con 0x7f4f1810ca40 2024-03-26T06:40:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.948+0000 7f4f21816740 1 Processor -- start 2024-03-26T06:40:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.948+0000 7f4f15ffb640 1 -- 172.21.15.38:0/3747675926 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f0800de80 con 0x7f4f1810ca40 2024-03-26T06:40:11.951 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.948+0000 7f4f21816740 1 -- start start 2024-03-26T06:40:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.949+0000 7f4f21816740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb9f20 0x560d85cb8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.949+0000 7f4f21816740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cb91b0 0x560d85cb9630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.949+0000 7f4f21816740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 0x560d85cc4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.949+0000 7f4f21816740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d85c1d5f0 con 0x560d85cb91b0 2024-03-26T06:40:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.949+0000 7f4f21816740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d85bc6930 con 0x560d85cc2730 2024-03-26T06:40:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.949+0000 7f4f21816740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d85c19690 con 0x560d85cb9f20 2024-03-26T06:40:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.949+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 0x560d85cc4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.949+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 0x560d85cc4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:55722/0 (socket says 172.21.15.38:55722) 2024-03-26T06:40:11.952 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.950+0000 7f4f17fff640 1 -- 172.21.15.38:0/436717149 learned_addr learned my addr 172.21.15.38:0/436717149 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.950+0000 7f4f17fff640 1 -- 172.21.15.38:0/436717149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb9f20 msgr2=0x560d85cb8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.950+0000 7f4f17fff640 1 --2- 172.21.15.38:0/436717149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb9f20 0x560d85cb8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.950+0000 7f4f17fff640 1 -- 172.21.15.38:0/436717149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cb91b0 msgr2=0x560d85cb9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:11.953 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.950+0000 7f4f17fff640 1 --2- 172.21.15.38:0/436717149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cb91b0 0x560d85cb9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.953+0000 7f4f17fff640 1 -- 172.21.15.38:0/436717149 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d85932210 con 0x560d85cc2730 2024-03-26T06:40:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.953+0000 7f4f15ffb640 1 -- 172.21.15.38:0/3747675926 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0800f620 con 0x7f4f1810ca40 2024-03-26T06:40:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.953+0000 7f4f17fff640 1 --2- 172.21.15.38:0/436717149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 0x560d85cc4b20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00a7d0 tx=0x7f4f0c00c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c7a1535f42584f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.954+0000 7f4f15ffb640 1 -- 172.21.15.38:0/3747675926 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4f0800f800 con 0x7f4f1810ca40 2024-03-26T06:40:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.954+0000 7f4efeffd640 1 -- 172.21.15.38:0/436717149 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c009bd0 con 0x560d85cc2730 2024-03-26T06:40:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.954+0000 7f4efeffd640 1 -- 172.21.15.38:0/436717149 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f0c009d70 con 0x560d85cc2730 2024-03-26T06:40:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.955+0000 7f4f21816740 1 -- 172.21.15.38:0/436717149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 msgr2=0x560d85cc4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.955+0000 7f4f21816740 1 --2- 172.21.15.38:0/436717149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 0x560d85cc4b20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00a7d0 tx=0x7f4f0c00c040 comp rx=0 tx=0).stop 2024-03-26T06:40:11.959 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.956+0000 7f4f15ffb640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f4f000a2990 0x7f4f000a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:11.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.957+0000 7f4f15ffb640 1 -- 172.21.15.38:0/3747675926 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f4f080cf9b0 con 0x7f4f1810ca40 2024-03-26T06:40:11.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.958+0000 7f4f1d0fd640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f4f000a2990 0x7f4f000a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.960+0000 7f4f1d0fd640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f4f000a2990 0x7f4f000a4e50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4f10006010 tx=0x7f4f10009840 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.964+0000 7f4f21816740 1 -- 172.21.15.38:0/436717149 shutdown_connections 2024-03-26T06:40:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.964+0000 7f4f21816740 1 --2- 172.21.15.38:0/436717149 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 0x560d85cc4b20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.964+0000 7f4f21816740 1 --2- 172.21.15.38:0/436717149 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cb91b0 0x560d85cb9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.964+0000 7f4f21816740 1 --2- 172.21.15.38:0/436717149 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb9f20 0x560d85cb8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.964+0000 7f4f21816740 1 -- 172.21.15.38:0/436717149 >> 172.21.15.38:0/436717149 conn(0x560d85c805b0 msgr2=0x560d85cb3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.964+0000 7f4f21816740 1 -- 172.21.15.38:0/436717149 shutdown_connections 2024-03-26T06:40:11.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.964+0000 7f4f21816740 1 -- 172.21.15.38:0/436717149 wait complete. 2024-03-26T06:40:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.964+0000 7f4f21816740 1 -- start start 2024-03-26T06:40:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.965+0000 7f4f21816740 1 Processor -- start 2024-03-26T06:40:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.965+0000 7f4f21816740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 0x560d85cbf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.965+0000 7f4f21816740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb9f20 0x560d85cbfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.965+0000 7f4f21816740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cc2730 0x560d85cc0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.965+0000 7f4f21816740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d85cc54b0 con 0x560d85cc2730 2024-03-26T06:40:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.965+0000 7f4f21816740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d85bd9d90 con 0x560d85cb9f20 2024-03-26T06:40:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.965+0000 7f4f21816740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d85cc5630 con 0x560d85cb91b0 2024-03-26T06:40:11.968 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.965+0000 7f4f21816740 10 client.0 _ll_register_callbacks cb 0x560d85931a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:40:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.966+0000 7f4f1d0fd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cc2730 0x560d85cc0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.966+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb9f20 0x560d85cbfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.966+0000 7f4f17fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb9f20 0x560d85cbfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:55724/0 (socket says 172.21.15.38:55724) 2024-03-26T06:40:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.966+0000 7f4f17fff640 1 -- 172.21.15.38:0/1169320646 learned_addr learned my addr 172.21.15.38:0/1169320646 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:11.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.966+0000 7f4f1c8fc640 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 0x560d85cbf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.967+0000 7f4f17fff640 1 -- 172.21.15.38:0/1169320646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 msgr2=0x560d85cbf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.967+0000 7f4f17fff640 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 0x560d85cbf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.967+0000 7f4f17fff640 1 -- 172.21.15.38:0/1169320646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cc2730 msgr2=0x560d85cc0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.967+0000 7f4f17fff640 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cc2730 0x560d85cc0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.967+0000 7f4f17fff640 1 -- 172.21.15.38:0/1169320646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560d859fe620 con 0x560d85cb9f20 2024-03-26T06:40:11.970 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.967+0000 7f4f17fff640 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb9f20 0x560d85cbfbc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00c010 tx=0x7f4f0c004300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f95ab3537e1885 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:11.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.968+0000 7f4efc7f8640 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-03-26T06:40:11.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.968+0000 7f4efc7f8640 1 -- 172.21.15.38:0/1169320646 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c025070 con 0x560d85cb9f20 2024-03-26T06:40:11.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.969+0000 7f4f21816740 1 -- 172.21.15.38:0/1169320646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f0c009d70 con 0x560d85cb9f20 2024-03-26T06:40:11.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.969+0000 7f4efc7f8640 1 -- 172.21.15.38:0/1169320646 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f4f0c0035f0 con 0x560d85cb9f20 2024-03-26T06:40:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.970+0000 7f4efc7f8640 1 -- 172.21.15.38:0/1169320646 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f0c02f050 con 0x560d85cb9f20 2024-03-26T06:40:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.970+0000 7f4f21816740 10 client.25199 fetch_fsmap learned FSMap version 27 2024-03-26T06:40:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.970+0000 7f4f21816740 1 -- 172.21.15.38:0/1169320646 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x560d85cd1580 con 0x560d85cb9f20 2024-03-26T06:40:11.973 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.971+0000 7f4efc7f8640 1 -- 172.21.15.38:0/1169320646 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f0c009d70 con 0x560d85cb9f20 2024-03-26T06:40:11.974 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.972+0000 7f4efc7f8640 1 -- 172.21.15.38:0/1169320646 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4f0c032020 con 0x560d85cb9f20 2024-03-26T06:40:11.975 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.972+0000 7f4f21816740 10 client.25199 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:40:11.975 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.972+0000 7f4f21816740 10 client.25199 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:40:11.975 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.972+0000 7f4f21816740 20 client.25199 populate_metadata read hostname 'smithi038' 2024-03-26T06:40:11.975 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.972+0000 7f4f21816740 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560d85cd2e60 0x560d85cd5280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:11.975 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.972+0000 7f4f21816740 4 client.25199 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:40:11.975 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.972+0000 7f4f21816740 1 -- 172.21.15.38:0/1169320646 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560d85c2ab30 con 0x560d85cd2e60 2024-03-26T06:40:11.976 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.973+0000 7f4f1c8fc640 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560d85cd2e60 0x560d85cd5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:11.978 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.976+0000 7f4f1c8fc640 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560d85cd2e60 0x560d85cd5280 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=758e623f5770a5af server_cookie=9a371370c2e0db65 in_seq=0 out_seq=0 2024-03-26T06:40:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.976+0000 7f4efc7f8640 10 client.25199 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.977+0000 7f4efc7f8640 1 -- 172.21.15.38:0/1169320646 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560d85c2ab30 con 0x560d85cd2e60 2024-03-26T06:40:11.979 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.977+0000 7f4efc7f8640 10 client.25199 handle_command_reply: tid=0 2024-03-26T06:40:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.977+0000 7f4f21816740 1 client.25199 shutdown 2024-03-26T06:40:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.977+0000 7f4f21816740 1 -- 172.21.15.38:0/1169320646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb9f20 msgr2=0x560d85cbfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:11.980 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.977+0000 7f4f21816740 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb9f20 0x560d85cbfbc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c00c010 tx=0x7f4f0c004300 comp rx=0 tx=0).stop 2024-03-26T06:40:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.977+0000 7f4f21816740 1 -- 172.21.15.38:0/1169320646 shutdown_connections 2024-03-26T06:40:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.977+0000 7f4f21816740 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560d85cd2e60 0x560d85cd5280 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.977+0000 7f4f21816740 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cc2730 0x560d85cc0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.977+0000 7f4f21816740 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb9f20 0x560d85cbfbc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.977+0000 7f4f21816740 1 --2- 172.21.15.38:0/1169320646 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 0x560d85cbf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.977+0000 7f4f21816740 1 -- 172.21.15.38:0/1169320646 >> 172.21.15.38:0/1169320646 conn(0x560d85c805b0 msgr2=0x560d85cb5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.978+0000 7f4efc7f8640 0 client.25199 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.978+0000 7f4f21816740 1 -- 172.21.15.38:0/1169320646 shutdown_connections 2024-03-26T06:40:11.981 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.978+0000 7f4f21816740 1 -- 172.21.15.38:0/1169320646 wait complete. 2024-03-26T06:40:11.985 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:11.978+0000 7f4f21816740 20 client.25199 trim_cache size 0 max 16384 2024-03-26T06:40:12.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.010+0000 7f4f21816740 1 Processor -- start 2024-03-26T06:40:12.012 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.010+0000 7f4f21816740 1 -- start start 2024-03-26T06:40:12.013 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.010+0000 7f4f21816740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 0x560d85cc1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.015+0000 7f4f1c8fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 0x560d85cc1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.015+0000 7f4f21816740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cb9f20 0x560d85cbff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.015+0000 7f4f1c8fc640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 0x560d85cc1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:58438/0 (socket says 172.21.15.38:58438) 2024-03-26T06:40:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.015+0000 7f4f1c8fc640 1 -- 172.21.15.38:0/2584106654 learned_addr learned my addr 172.21.15.38:0/2584106654 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:12.018 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.015+0000 7f4f21816740 1 --2- 172.21.15.38:0/2584106654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 0x560d85cbc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.015+0000 7f4f21816740 1 -- 172.21.15.38:0/2584106654 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f04010a30 con 0x560d85cb9f20 2024-03-26T06:40:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.016+0000 7f4f21816740 1 -- 172.21.15.38:0/2584106654 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0400e570 con 0x560d85cc2730 2024-03-26T06:40:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.016+0000 7f4f1d0fd640 1 --2- 172.21.15.38:0/2584106654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 0x560d85cbc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.016+0000 7f4f21816740 1 -- 172.21.15.38:0/2584106654 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0400c6c0 con 0x560d85cb91b0 2024-03-26T06:40:12.019 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.016+0000 7f4f17fff640 1 --2- 172.21.15.38:0/2584106654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cb9f20 0x560d85cbff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.016+0000 7f4f1c8fc640 1 -- 172.21.15.38:0/2584106654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 msgr2=0x560d85cbc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.016+0000 7f4f1c8fc640 1 --2- 172.21.15.38:0/2584106654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 0x560d85cbc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.016+0000 7f4f1c8fc640 1 -- 172.21.15.38:0/2584106654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cb9f20 msgr2=0x560d85cbff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.016+0000 7f4f1c8fc640 1 --2- 172.21.15.38:0/2584106654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cb9f20 0x560d85cbff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.017+0000 7f4f1c8fc640 1 -- 172.21.15.38:0/2584106654 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d85cb8fd0 con 0x560d85cb91b0 2024-03-26T06:40:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.017+0000 7f4f1c8fc640 1 --2- 172.21.15.38:0/2584106654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 0x560d85cc1d40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4f080989f0 tx=0x7f4f080bc040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c591d363bb566068 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:12.020 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.017+0000 7f4efe7fc640 1 -- 172.21.15.38:0/2584106654 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f080b4070 con 0x560d85cb91b0 2024-03-26T06:40:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.018+0000 7f4efe7fc640 1 -- 172.21.15.38:0/2584106654 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f080b7020 con 0x560d85cb91b0 2024-03-26T06:40:12.021 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.019+0000 7f4f21816740 1 -- 172.21.15.38:0/2584106654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 msgr2=0x560d85cc1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.019+0000 7f4f21816740 1 --2- 172.21.15.38:0/2584106654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 0x560d85cc1d40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4f080989f0 tx=0x7f4f080bc040 comp rx=0 tx=0).stop 2024-03-26T06:40:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.019+0000 7f4f21816740 1 -- 172.21.15.38:0/2584106654 shutdown_connections 2024-03-26T06:40:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.019+0000 7f4f21816740 1 --2- 172.21.15.38:0/2584106654 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cc2730 0x560d85cbc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.019+0000 7f4f21816740 1 --2- 172.21.15.38:0/2584106654 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cb9f20 0x560d85cbff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.019+0000 7f4f21816740 1 --2- 172.21.15.38:0/2584106654 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb91b0 0x560d85cc1d40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.019+0000 7f4f21816740 1 -- 172.21.15.38:0/2584106654 >> 172.21.15.38:0/2584106654 conn(0x560d85c805b0 msgr2=0x560d85cd9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.019+0000 7f4f21816740 1 -- 172.21.15.38:0/2584106654 shutdown_connections 2024-03-26T06:40:12.022 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.019+0000 7f4f21816740 1 -- 172.21.15.38:0/2584106654 wait complete. 2024-03-26T06:40:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.020+0000 7f4f21816740 1 -- start start 2024-03-26T06:40:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.020+0000 7f4f21816740 1 Processor -- start 2024-03-26T06:40:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.020+0000 7f4f21816740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb91b0 0x560d85cc7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.020+0000 7f4f21816740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb9f20 0x560d85cc7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.020+0000 7f4f21816740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cc2730 0x560d85cc8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.020+0000 7f4f21816740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0400cc30 con 0x560d85cc2730 2024-03-26T06:40:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.020+0000 7f4f21816740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0400df10 con 0x560d85cb91b0 2024-03-26T06:40:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.020+0000 7f4f21816740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0400d250 con 0x560d85cb9f20 2024-03-26T06:40:12.023 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.020+0000 7f4f21816740 10 client.0 _ll_register_callbacks cb 0x7f4f0c00a3d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:40:12.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.022+0000 7f4f1c8fc640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb91b0 0x560d85cc7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:12.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.022+0000 7f4f1d0fd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cc2730 0x560d85cc8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:12.024 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.022+0000 7f4f1d0fd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cc2730 0x560d85cc8000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40348/0 (socket says 172.21.15.38:40348) 2024-03-26T06:40:12.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.022+0000 7f4f1d0fd640 1 -- 172.21.15.38:0/512122586 learned_addr learned my addr 172.21.15.38:0/512122586 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:12.025 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.022+0000 7f4f17fff640 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb9f20 0x560d85cc7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:12.026 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.023+0000 7f4f17fff640 1 -- 172.21.15.38:0/512122586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb91b0 msgr2=0x560d85cc7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:12.026 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.023+0000 7f4f17fff640 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb91b0 0x560d85cc7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:12.026 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.023+0000 7f4f17fff640 1 -- 172.21.15.38:0/512122586 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cc2730 msgr2=0x560d85cc8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:12.026 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.023+0000 7f4f17fff640 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cc2730 0x560d85cc8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:12.026 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.023+0000 7f4f17fff640 1 -- 172.21.15.38:0/512122586 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560d85d28560 con 0x560d85cb9f20 2024-03-26T06:40:12.027 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.024+0000 7f4f17fff640 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb9f20 0x560d85cc7ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0054c0 tx=0x7f4f0c00a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=458f94f714bcc259 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:12.027 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.024+0000 7f4efc7f8640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-26T06:40:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.024+0000 7f4efc7f8640 1 -- 172.21.15.38:0/512122586 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0c038060 con 0x560d85cb9f20 2024-03-26T06:40:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.024+0000 7f4f21816740 1 -- 172.21.15.38:0/512122586 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f080b7020 con 0x560d85cb9f20 2024-03-26T06:40:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.025+0000 7f4efc7f8640 1 -- 172.21.15.38:0/512122586 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f4f0c016b40 con 0x560d85cb9f20 2024-03-26T06:40:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.025+0000 7f4efc7f8640 1 -- 172.21.15.38:0/512122586 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f0c025040 con 0x560d85cb9f20 2024-03-26T06:40:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.025+0000 7f4f21816740 10 client.25093 fetch_fsmap learned FSMap version 27 2024-03-26T06:40:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.025+0000 7f4f21816740 1 -- 172.21.15.38:0/512122586 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x560d85d29850 con 0x560d85cb9f20 2024-03-26T06:40:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.025+0000 7f4efc7f8640 1 -- 172.21.15.38:0/512122586 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f080b7020 con 0x560d85cb9f20 2024-03-26T06:40:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.026+0000 7f4efc7f8640 1 -- 172.21.15.38:0/512122586 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f4f0c032070 con 0x560d85cb9f20 2024-03-26T06:40:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.026+0000 7f4f21816740 10 client.25093 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:40:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.026+0000 7f4f21816740 10 client.25093 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:40:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.026+0000 7f4f21816740 20 client.25093 populate_metadata read hostname 'smithi038' 2024-03-26T06:40:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.026+0000 7f4f21816740 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560d85d29e40 0x560d85d2c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.026+0000 7f4f21816740 4 client.25093 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:40:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.026+0000 7f4f21816740 1 -- 172.21.15.38:0/512122586 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560d85c24e80 con 0x560d85d29e40 2024-03-26T06:40:12.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.026+0000 7f4f1c8fc640 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560d85d29e40 0x560d85d2c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:12.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.027+0000 7f4f1c8fc640 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560d85d29e40 0x560d85d2c260 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97a2c96622201237 server_cookie=68fdd5583d90399 in_seq=0 out_seq=0 2024-03-26T06:40:12.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:12.027+0000 7f4efc7f8640 10 client.25093 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:12.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:12 smithi038 ceph-mon[32265]: pgmap v83: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 651 KiB/s rd, 325 MiB/s wr, 32.16k op/s 2024-03-26T06:40:12.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:12 smithi121 ceph-mon[36349]: pgmap v83: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 651 KiB/s rd, 325 MiB/s wr, 32.16k op/s 2024-03-26T06:40:12.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:12 smithi170 ceph-mon[36912]: pgmap v83: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1013 GiB / 1.0 TiB avail; 651 KiB/s rd, 325 MiB/s wr, 32.16k op/s 2024-03-26T06:40:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4efc7f8640 1 -- 172.21.15.38:0/512122586 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560d85c24e80 con 0x560d85d29e40 2024-03-26T06:40:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4efc7f8640 10 client.25093 handle_command_reply: tid=0 2024-03-26T06:40:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4f21816740 1 client.25093 shutdown 2024-03-26T06:40:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4f21816740 1 -- 172.21.15.38:0/512122586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb9f20 msgr2=0x560d85cc7ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4f21816740 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb9f20 0x560d85cc7ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4f0c0054c0 tx=0x7f4f0c00a6a0 comp rx=0 tx=0).stop 2024-03-26T06:40:13.778 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4f21816740 1 -- 172.21.15.38:0/512122586 shutdown_connections 2024-03-26T06:40:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4f21816740 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x560d85d29e40 0x560d85d2c260 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4f21816740 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560d85cc2730 0x560d85cc8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4efc7f8640 0 client.25093 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4f21816740 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x560d85cb9f20 0x560d85cc7ac0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4f21816740 1 --2- 172.21.15.38:0/512122586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x560d85cb91b0 0x560d85cc7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4f21816740 1 -- 172.21.15.38:0/512122586 >> 172.21.15.38:0/512122586 conn(0x560d85c805b0 msgr2=0x560d85cbdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.775+0000 7f4f21816740 1 -- 172.21.15.38:0/512122586 shutdown_connections 2024-03-26T06:40:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.776+0000 7f4f21816740 1 -- 172.21.15.38:0/512122586 wait complete. 2024-03-26T06:40:13.779 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.776+0000 7f4f21816740 20 client.25093 trim_cache size 0 max 16384 2024-03-26T06:40:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:40:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:40:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:40:13.780 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:40:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.777+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3747675926 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f4f000a2990 msgr2=0x7f4f000a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.777+0000 7f4f1d8fe640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f4f000a2990 0x7f4f000a4e50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4f10006010 tx=0x7f4f10009840 comp rx=0 tx=0).stop 2024-03-26T06:40:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.777+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3747675926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f1810ca40 msgr2=0x7f4f18083fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.777+0000 7f4f1d8fe640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f1810ca40 0x7f4f18083fb0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4f0800d950 tx=0x7f4f0800dd50 comp rx=0 tx=0).stop 2024-03-26T06:40:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.777+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3747675926 shutdown_connections 2024-03-26T06:40:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.777+0000 7f4f1d8fe640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f4f000a2990 0x7f4f000a4e50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:13.781 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.777+0000 7f4f1d8fe640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4f1810df60 0x7f4f1807e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:13.782 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.777+0000 7f4f1d8fe640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4f1810d640 0x7f4f1807e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:13.782 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.778+0000 7f4f1d8fe640 1 --2- 172.21.15.38:0/3747675926 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f4f1810ca40 0x7f4f18083fb0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:13.782 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.778+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3747675926 >> 172.21.15.38:0/3747675926 conn(0x7f4f1806c200 msgr2=0x7f4f18071d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:13.782 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.778+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3747675926 shutdown_connections 2024-03-26T06:40:13.782 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:13.778+0000 7f4f1d8fe640 1 -- 172.21.15.38:0/3747675926 wait complete. 2024-03-26T06:40:14.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:14 smithi038 ceph-mon[32265]: pgmap v84: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1012 GiB / 1.0 TiB avail; 256 KiB/s rd, 206 MiB/s wr, 21.84k op/s 2024-03-26T06:40:14.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:14 smithi121 ceph-mon[36349]: pgmap v84: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1012 GiB / 1.0 TiB avail; 256 KiB/s rd, 206 MiB/s wr, 21.84k op/s 2024-03-26T06:40:14.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:14 smithi170 ceph-mon[36912]: pgmap v84: 97 pgs: 97 active+clean; 13 GiB data, 60 GiB used, 1012 GiB / 1.0 TiB avail; 256 KiB/s rd, 206 MiB/s wr, 21.84k op/s 2024-03-26T06:40:16.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:16 smithi038 ceph-mon[32265]: pgmap v85: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 298 KiB/s rd, 228 MiB/s wr, 25.68k op/s 2024-03-26T06:40:16.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:16 smithi121 ceph-mon[36349]: pgmap v85: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 298 KiB/s rd, 228 MiB/s wr, 25.68k op/s 2024-03-26T06:40:16.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:16 smithi170 ceph-mon[36912]: pgmap v85: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 298 KiB/s rd, 228 MiB/s wr, 25.68k op/s 2024-03-26T06:40:18.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:18 smithi038 ceph-mon[32265]: pgmap v86: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 141 KiB/s rd, 175 MiB/s wr, 21.78k op/s 2024-03-26T06:40:18.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:18 smithi121 ceph-mon[36349]: pgmap v86: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 141 KiB/s rd, 175 MiB/s wr, 21.78k op/s 2024-03-26T06:40:18.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:18 smithi170 ceph-mon[36912]: pgmap v86: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 141 KiB/s rd, 175 MiB/s wr, 21.78k op/s 2024-03-26T06:40:20.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:20 smithi038 ceph-mon[32265]: pgmap v87: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 141 KiB/s rd, 128 MiB/s wr, 17.27k op/s 2024-03-26T06:40:20.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:20 smithi121 ceph-mon[36349]: pgmap v87: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 141 KiB/s rd, 128 MiB/s wr, 17.27k op/s 2024-03-26T06:40:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:20 smithi170 ceph-mon[36912]: pgmap v87: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 141 KiB/s rd, 128 MiB/s wr, 17.27k op/s 2024-03-26T06:40:22.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:22 smithi038 ceph-mon[32265]: pgmap v88: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 161 KiB/s rd, 179 MiB/s wr, 24.97k op/s 2024-03-26T06:40:22.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:22 smithi121 ceph-mon[36349]: pgmap v88: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 161 KiB/s rd, 179 MiB/s wr, 24.97k op/s 2024-03-26T06:40:22.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:22 smithi170 ceph-mon[36912]: pgmap v88: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 161 KiB/s rd, 179 MiB/s wr, 24.97k op/s 2024-03-26T06:40:24.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:24 smithi038 ceph-mon[32265]: pgmap v89: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1009 GiB / 1.0 TiB avail; 63 KiB/s rd, 107 MiB/s wr, 16.69k op/s 2024-03-26T06:40:24.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:24 smithi121 ceph-mon[36349]: pgmap v89: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1009 GiB / 1.0 TiB avail; 63 KiB/s rd, 107 MiB/s wr, 16.69k op/s 2024-03-26T06:40:24.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:24 smithi170 ceph-mon[36912]: pgmap v89: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1009 GiB / 1.0 TiB avail; 63 KiB/s rd, 107 MiB/s wr, 16.69k op/s 2024-03-26T06:40:26.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:26 smithi038 ceph-mon[32265]: pgmap v90: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 85 KiB/s rd, 116 MiB/s wr, 19.31k op/s 2024-03-26T06:40:26.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:26 smithi121 ceph-mon[36349]: pgmap v90: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 85 KiB/s rd, 116 MiB/s wr, 19.31k op/s 2024-03-26T06:40:26.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:26 smithi170 ceph-mon[36912]: pgmap v90: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 85 KiB/s rd, 116 MiB/s wr, 19.31k op/s 2024-03-26T06:40:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.948+0000 7fa79a7d4640 1 -- 172.21.15.38:0/2428561684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa79410fea0 msgr2=0x7fa79410db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:26.952 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.948+0000 7fa79a7d4640 1 --2- 172.21.15.38:0/2428561684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa79410fea0 0x7fa79410db90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa78c01acb0 tx=0x7fa78c03d170 comp rx=0 tx=0).stop 2024-03-26T06:40:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.948+0000 7fa79a7d4640 1 -- 172.21.15.38:0/2428561684 shutdown_connections 2024-03-26T06:40:26.953 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.948+0000 7fa79a7d4640 1 --2- 172.21.15.38:0/2428561684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa79411aad0 0x7fa79411cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.948+0000 7fa79a7d4640 1 --2- 172.21.15.38:0/2428561684 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa79410fea0 0x7fa79410db90 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.948+0000 7fa79a7d4640 1 --2- 172.21.15.38:0/2428561684 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79410fa30 0x7fa79410d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.948+0000 7fa79a7d4640 1 -- 172.21.15.38:0/2428561684 >> 172.21.15.38:0/2428561684 conn(0x7fa79406c390 msgr2=0x7fa79406c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.949+0000 7fa79a7d4640 1 -- 172.21.15.38:0/2428561684 shutdown_connections 2024-03-26T06:40:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.949+0000 7fa79a7d4640 1 -- 172.21.15.38:0/2428561684 wait complete. 2024-03-26T06:40:26.954 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.950+0000 7fa79a7d4640 1 Processor -- start 2024-03-26T06:40:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.950+0000 7fa79a7d4640 1 -- start start 2024-03-26T06:40:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.951+0000 7fa79a7d4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79410fa30 0x7fa79411a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.951+0000 7fa79a7d4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa79410fea0 0x7fa794116310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.951+0000 7fa79a7d4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa79411aad0 0x7fa794116850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.951+0000 7fa79a7d4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa794112d20 con 0x7fa79411aad0 2024-03-26T06:40:26.955 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.951+0000 7fa79a7d4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa794112ba0 con 0x7fa79410fa30 2024-03-26T06:40:26.956 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.951+0000 7fa79a7d4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa794112ea0 con 0x7fa79410fea0 2024-03-26T06:40:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.951+0000 7fa798d71640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa79411aad0 0x7fa794116850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.952+0000 7fa793fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79410fa30 0x7fa79411a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.952+0000 7fa7937fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa79410fea0 0x7fa794116310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.952+0000 7fa798d71640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa79411aad0 0x7fa794116850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:43030/0 (socket says 172.21.15.38:43030) 2024-03-26T06:40:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.952+0000 7fa798d71640 1 -- 172.21.15.38:0/878934729 learned_addr learned my addr 172.21.15.38:0/878934729 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.952+0000 7fa798d71640 1 -- 172.21.15.38:0/878934729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa79410fea0 msgr2=0x7fa794116310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.952+0000 7fa798d71640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa79410fea0 0x7fa794116310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.952+0000 7fa798d71640 1 -- 172.21.15.38:0/878934729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79410fa30 msgr2=0x7fa79411a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.952+0000 7fa798d71640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79410fa30 0x7fa79411a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.952+0000 7fa798d71640 1 -- 172.21.15.38:0/878934729 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa794116fd0 con 0x7fa79411aad0 2024-03-26T06:40:26.957 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.953+0000 7fa798d71640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa79411aad0 0x7fa794116850 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa78400b3c0 tx=0x7fa78400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.953+0000 7fa7917fa640 1 -- 172.21.15.38:0/878934729 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784011830 con 0x7fa79411aad0 2024-03-26T06:40:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.954+0000 7fa79a7d4640 1 -- 172.21.15.38:0/878934729 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7941aa870 con 0x7fa79411aad0 2024-03-26T06:40:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.954+0000 7fa79a7d4640 1 -- 172.21.15.38:0/878934729 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7941aad80 con 0x7fa79411aad0 2024-03-26T06:40:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.954+0000 7fa7917fa640 1 -- 172.21.15.38:0/878934729 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa784016070 con 0x7fa79411aad0 2024-03-26T06:40:26.958 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.956+0000 7fa7917fa640 1 -- 172.21.15.38:0/878934729 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784010380 con 0x7fa79411aad0 2024-03-26T06:40:26.960 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.958+0000 7fa79d3b9740 1 Processor -- start 2024-03-26T06:40:26.961 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.958+0000 7fa79d3b9740 1 -- start start 2024-03-26T06:40:26.961 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.958+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 0x5601d5356a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:26.961 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.958+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5356fd0 0x5601d5357450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:26.962 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.958+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d5360550 0x5601d5362940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:26.962 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.958+0000 7fa79d3b9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d52bb5f0 con 0x5601d5360550 2024-03-26T06:40:26.962 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.958+0000 7fa79d3b9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d5264930 con 0x5601d5356fd0 2024-03-26T06:40:26.962 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.958+0000 7fa79d3b9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d52b7690 con 0x5601d5357d40 2024-03-26T06:40:26.962 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.959+0000 7fa7937fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 0x5601d5356a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:26.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.959+0000 7fa7937fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 0x5601d5356a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:57862/0 (socket says 172.21.15.38:57862) 2024-03-26T06:40:26.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.959+0000 7fa7937fe640 1 -- 172.21.15.38:0/1852395470 learned_addr learned my addr 172.21.15.38:0/1852395470 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:26.963 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.960+0000 7fa793fff640 1 --2- 172.21.15.38:0/1852395470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5356fd0 0x5601d5357450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.961+0000 7fa793fff640 1 -- 172.21.15.38:0/1852395470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 msgr2=0x5601d5356a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.961+0000 7fa793fff640 1 --2- 172.21.15.38:0/1852395470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 0x5601d5356a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.961+0000 7fa793fff640 1 -- 172.21.15.38:0/1852395470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d5360550 msgr2=0x5601d5362940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.961+0000 7fa793fff640 1 --2- 172.21.15.38:0/1852395470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d5360550 0x5601d5362940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.961+0000 7fa793fff640 1 -- 172.21.15.38:0/1852395470 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601d4fd0210 con 0x5601d5356fd0 2024-03-26T06:40:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.963+0000 7fa7917fa640 1 -- 172.21.15.38:0/878934729 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa784010520 con 0x7fa79411aad0 2024-03-26T06:40:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.964+0000 7fa7917fa640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa77c0a2a10 0x7fa77c0a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.965+0000 7fa793fff640 1 --2- 172.21.15.38:0/1852395470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5356fd0 0x5601d5357450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa78800a800 tx=0x7fa78800acc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d95a94512e180380 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.967+0000 7fa7937fe640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa77c0a2a10 0x7fa77c0a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.967+0000 7fa7917fa640 1 -- 172.21.15.38:0/878934729 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fa78401b760 con 0x7fa79411aad0 2024-03-26T06:40:26.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.969+0000 7fa77a7fc640 1 -- 172.21.15.38:0/1852395470 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788015070 con 0x5601d5356fd0 2024-03-26T06:40:26.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.969+0000 7fa7937fe640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa77c0a2a10 0x7fa77c0a4ed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x5601d5357a70 tx=0x7fa78c03f040 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:26.972 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.969+0000 7fa77a7fc640 1 -- 172.21.15.38:0/1852395470 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa78800cb40 con 0x5601d5356fd0 2024-03-26T06:40:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.970+0000 7fa79d3b9740 1 -- 172.21.15.38:0/1852395470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5356fd0 msgr2=0x5601d5357450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.970+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/1852395470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5356fd0 0x5601d5357450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa78800a800 tx=0x7fa78800acc0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.971+0000 7fa79d3b9740 1 -- 172.21.15.38:0/1852395470 shutdown_connections 2024-03-26T06:40:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.971+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/1852395470 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d5360550 0x5601d5362940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.971+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/1852395470 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5356fd0 0x5601d5357450 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.971+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/1852395470 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 0x5601d5356a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.974 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.971+0000 7fa79d3b9740 1 -- 172.21.15.38:0/1852395470 >> 172.21.15.38:0/1852395470 conn(0x5601d531e5b0 msgr2=0x5601d5351cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:26.976 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.973+0000 7fa79d3b9740 1 -- 172.21.15.38:0/1852395470 shutdown_connections 2024-03-26T06:40:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.973+0000 7fa79d3b9740 1 -- 172.21.15.38:0/1852395470 wait complete. 2024-03-26T06:40:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.974+0000 7fa79d3b9740 1 -- start start 2024-03-26T06:40:26.977 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.974+0000 7fa79d3b9740 1 Processor -- start 2024-03-26T06:40:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.975+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 0x5601d535fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.975+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5360550 0x5601d535d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.975+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d535d6c0 0x5601d535db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.975+0000 7fa79d3b9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d53632d0 con 0x5601d535d6c0 2024-03-26T06:40:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.975+0000 7fa79d3b9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d5277d90 con 0x5601d5360550 2024-03-26T06:40:26.983 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.975+0000 7fa79d3b9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d5363450 con 0x5601d5357d40 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.975+0000 7fa79d3b9740 10 client.0 _ll_register_callbacks cb 0x5601d4fcfa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.975+0000 7fa793fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 0x5601d535fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.975+0000 7fa793fff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 0x5601d535fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:57864/0 (socket says 172.21.15.38:57864) 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.976+0000 7fa793fff640 1 -- 172.21.15.38:0/3452917866 learned_addr learned my addr 172.21.15.38:0/3452917866 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.976+0000 7fa798d71640 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d535d6c0 0x5601d535db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.976+0000 7fa7937fe640 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5360550 0x5601d535d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.976+0000 7fa793fff640 1 -- 172.21.15.38:0/3452917866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5360550 msgr2=0x5601d535d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.976+0000 7fa793fff640 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5360550 0x5601d535d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.976+0000 7fa793fff640 1 -- 172.21.15.38:0/3452917866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d535d6c0 msgr2=0x5601d535db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.976+0000 7fa793fff640 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d535d6c0 0x5601d535db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.976+0000 7fa793fff640 1 -- 172.21.15.38:0/3452917866 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601d509c620 con 0x5601d5357d40 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.977+0000 7fa793fff640 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 0x5601d535fe90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa788008010 tx=0x7fa78800a310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0b400c1706ea300 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.977+0000 7fa7737fe640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-26T06:40:26.984 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.977+0000 7fa7737fe640 1 -- 172.21.15.38:0/3452917866 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788027060 con 0x5601d5357d40 2024-03-26T06:40:26.985 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.978+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3452917866 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5601d52c8b30 con 0x5601d5357d40 2024-03-26T06:40:26.985 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.978+0000 7fa7737fe640 1 -- 172.21.15.38:0/3452917866 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fa788004450 con 0x5601d5357d40 2024-03-26T06:40:26.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.980+0000 7fa7737fe640 1 -- 172.21.15.38:0/3452917866 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa788015050 con 0x5601d5357d40 2024-03-26T06:40:26.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.980+0000 7fa79d3b9740 10 client.25102 fetch_fsmap learned FSMap version 27 2024-03-26T06:40:26.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.980+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3452917866 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5601d536f760 con 0x5601d5357d40 2024-03-26T06:40:26.987 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.984+0000 7fa7737fe640 1 -- 172.21.15.38:0/3452917866 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5601d52c8b30 con 0x5601d5357d40 2024-03-26T06:40:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.985+0000 7fa7737fe640 1 -- 172.21.15.38:0/3452917866 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fa788021030 con 0x5601d5357d40 2024-03-26T06:40:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.985+0000 7fa79d3b9740 10 client.25102 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:40:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.986+0000 7fa79d3b9740 10 client.25102 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:40:26.991 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.986+0000 7fa79d3b9740 20 client.25102 populate_metadata read hostname 'smithi038' 2024-03-26T06:40:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.986+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5601d5370e60 0x5601d53732c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.986+0000 7fa79d3b9740 4 client.25102 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:40:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.986+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3452917866 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601d52c2e80 con 0x5601d5370e60 2024-03-26T06:40:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.987+0000 7fa7937fe640 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5601d5370e60 0x5601d53732c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.988+0000 7fa7937fe640 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5601d5370e60 0x5601d53732c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f696db77f9eda7c3 server_cookie=f617b669c536c735 in_seq=0 out_seq=0 2024-03-26T06:40:26.992 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.988+0000 7fa7737fe640 10 client.25102 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:27.001 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.998+0000 7fa7737fe640 1 -- 172.21.15.38:0/3452917866 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5601d52c2e80 con 0x5601d5370e60 2024-03-26T06:40:27.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.998+0000 7fa7737fe640 10 client.25102 handle_command_reply: tid=0 2024-03-26T06:40:27.002 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:26.998+0000 7fa79d3b9740 1 client.25102 shutdown 2024-03-26T06:40:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.002+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3452917866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 msgr2=0x5601d535fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:27.005 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.002+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 0x5601d535fe90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa788008010 tx=0x7fa78800a310 comp rx=0 tx=0).stop 2024-03-26T06:40:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.003+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3452917866 shutdown_connections 2024-03-26T06:40:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.003+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5601d5370e60 0x5601d53732c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.003+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d535d6c0 0x5601d535db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.003+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5360550 0x5601d535d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.003+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3452917866 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d5357d40 0x5601d535fe90 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa788008010 tx=0x7fa78800a310 comp rx=0 tx=0).stop 2024-03-26T06:40:27.006 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.003+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3452917866 >> 172.21.15.38:0/3452917866 conn(0x5601d531e5b0 msgr2=0x5601d5353180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.003+0000 7fa7737fe640 0 client.25102 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.003+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3452917866 shutdown_connections 2024-03-26T06:40:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.003+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3452917866 wait complete. 2024-03-26T06:40:27.008 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.004+0000 7fa79d3b9740 20 client.25102 trim_cache size 0 max 16384 2024-03-26T06:40:27.048 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.045+0000 7fa79d3b9740 1 Processor -- start 2024-03-26T06:40:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.045+0000 7fa79d3b9740 1 -- start start 2024-03-26T06:40:27.049 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.046+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d531e5b0 0x5601d5361940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.046+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d535f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.046+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5360550 0x5601d5358330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.047+0000 7fa79d3b9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa780010a00 con 0x5601d531e5b0 2024-03-26T06:40:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.047+0000 7fa7937fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d535f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.047+0000 7fa7937fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d535f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:57880/0 (socket says 172.21.15.38:57880) 2024-03-26T06:40:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.047+0000 7fa7937fe640 1 -- 172.21.15.38:0/3403015776 learned_addr learned my addr 172.21.15.38:0/3403015776 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.047+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3403015776 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78000e540 con 0x5601d5360550 2024-03-26T06:40:27.050 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.047+0000 7fa798d71640 1 --2- 172.21.15.38:0/3403015776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5360550 0x5601d5358330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:27.051 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.047+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3403015776 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78000c690 con 0x5601d535d6c0 2024-03-26T06:40:27.053 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.050+0000 7fa793fff640 1 --2- 172.21.15.38:0/3403015776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d531e5b0 0x5601d5361940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.052+0000 7fa7937fe640 1 -- 172.21.15.38:0/3403015776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5360550 msgr2=0x5601d5358330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:27.054 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.052+0000 7fa7937fe640 1 --2- 172.21.15.38:0/3403015776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5360550 0x5601d5358330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.052+0000 7fa7937fe640 1 -- 172.21.15.38:0/3403015776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d531e5b0 msgr2=0x5601d5361940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.052+0000 7fa7937fe640 1 --2- 172.21.15.38:0/3403015776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d531e5b0 0x5601d5361940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.052+0000 7fa7937fe640 1 -- 172.21.15.38:0/3403015776 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601d53562a0 con 0x5601d535d6c0 2024-03-26T06:40:27.055 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.052+0000 7fa7937fe640 1 --2- 172.21.15.38:0/3403015776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d535f9c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa78c01aa70 tx=0x7fa78c04f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1b38d94bf5df622 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.055+0000 7fa779ffb640 1 -- 172.21.15.38:0/3403015776 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78c067070 con 0x5601d535d6c0 2024-03-26T06:40:27.058 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.056+0000 7fa779ffb640 1 -- 172.21.15.38:0/3403015776 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa78c0616b0 con 0x5601d535d6c0 2024-03-26T06:40:27.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.057+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3403015776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 msgr2=0x5601d535f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:27.059 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.057+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3403015776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d535f9c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa78c01aa70 tx=0x7fa78c04f000 comp rx=0 tx=0).stop 2024-03-26T06:40:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.057+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3403015776 shutdown_connections 2024-03-26T06:40:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.057+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3403015776 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d5360550 0x5601d5358330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.058+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3403015776 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d535f9c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.058+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3403015776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d531e5b0 0x5601d5361940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:27.060 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.058+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3403015776 >> 172.21.15.38:0/3403015776 conn(0x5601d534d190 msgr2=0x5601d5377c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:27.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.059+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3403015776 shutdown_connections 2024-03-26T06:40:27.062 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.059+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3403015776 wait complete. 2024-03-26T06:40:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.059+0000 7fa79d3b9740 1 -- start start 2024-03-26T06:40:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.059+0000 7fa79d3b9740 1 Processor -- start 2024-03-26T06:40:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.060+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d531e5b0 0x5601d5363940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.060+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d5363e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:27.063 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.060+0000 7fa79d3b9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d5360550 0x5601d53c6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.061+0000 7fa793fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d531e5b0 0x5601d5363940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.061+0000 7fa793fff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d531e5b0 0x5601d5363940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:49452/0 (socket says 172.21.15.38:49452) 2024-03-26T06:40:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.061+0000 7fa7937fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d5363e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.061+0000 7fa7937fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d5363e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:57888/0 (socket says 172.21.15.38:57888) 2024-03-26T06:40:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.061+0000 7fa793fff640 1 -- 172.21.15.38:0/3800135242 learned_addr learned my addr 172.21.15.38:0/3800135242 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.061+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3800135242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78000cc00 con 0x5601d5360550 2024-03-26T06:40:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.061+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3800135242 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78000dee0 con 0x5601d531e5b0 2024-03-26T06:40:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.061+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3800135242 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78000d220 con 0x5601d535d6c0 2024-03-26T06:40:27.079 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.061+0000 7fa79d3b9740 10 client.0 _ll_register_callbacks cb 0x7fa788006f30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.062+0000 7fa798d71640 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d5360550 0x5601d53c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.063+0000 7fa798d71640 1 -- 172.21.15.38:0/3800135242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 msgr2=0x5601d5363e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.063+0000 7fa798d71640 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d5363e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.063+0000 7fa798d71640 1 -- 172.21.15.38:0/3800135242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d531e5b0 msgr2=0x5601d5363940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.063+0000 7fa798d71640 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d531e5b0 0x5601d5363940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.063+0000 7fa798d71640 1 -- 172.21.15.38:0/3800135242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601d5362980 con 0x5601d5360550 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.065+0000 7fa798d71640 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d5360550 0x5601d53c6390 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa7840cbb40 tx=0x7fa784083f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=446a0fc28cba627f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.065+0000 7fa7737fe640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.065+0000 7fa7737fe640 1 -- 172.21.15.38:0/3800135242 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7840cbeb0 con 0x5601d5360550 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.065+0000 7fa793fff640 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d531e5b0 0x5601d5363940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.066+0000 7fa7937fe640 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d5363e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.066+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3800135242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa78c0616b0 con 0x5601d5360550 2024-03-26T06:40:27.080 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.066+0000 7fa7737fe640 1 -- 172.21.15.38:0/3800135242 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fa7840b9120 con 0x5601d5360550 2024-03-26T06:40:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.066+0000 7fa7737fe640 1 -- 172.21.15.38:0/3800135242 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7840b9320 con 0x5601d5360550 2024-03-26T06:40:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.067+0000 7fa79d3b9740 10 client.15585 fetch_fsmap learned FSMap version 27 2024-03-26T06:40:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.067+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3800135242 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5601d53c79d0 con 0x5601d5360550 2024-03-26T06:40:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.067+0000 7fa7737fe640 1 -- 172.21.15.38:0/3800135242 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa78c0616b0 con 0x5601d5360550 2024-03-26T06:40:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.067+0000 7fa7737fe640 1 -- 172.21.15.38:0/3800135242 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fa7840969a0 con 0x5601d5360550 2024-03-26T06:40:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.068+0000 7fa79d3b9740 10 client.15585 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:40:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.068+0000 7fa79d3b9740 10 client.15585 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:40:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.068+0000 7fa79d3b9740 20 client.15585 populate_metadata read hostname 'smithi038' 2024-03-26T06:40:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.068+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5601d53c8230 0x5601d53ca650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.068+0000 7fa79d3b9740 4 client.15585 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:40:27.081 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.068+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3800135242 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5601d5241710 con 0x5601d53c8230 2024-03-26T06:40:27.082 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.077+0000 7fa793fff640 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5601d53c8230 0x5601d53ca650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:27.090 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.083+0000 7fa7737fe640 10 client.15585 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:27.091 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:27.083+0000 7fa793fff640 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5601d53c8230 0x5601d53ca650 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4b201fdd27d5a8e server_cookie=bc53f98ae8fb118c in_seq=0 out_seq=0 2024-03-26T06:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.436+0000 7fa7737fe640 1 -- 172.21.15.38:0/3800135242 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5601d5241710 con 0x5601d53c8230 2024-03-26T06:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.436+0000 7fa7737fe640 10 client.15585 handle_command_reply: tid=0 2024-03-26T06:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.436+0000 7fa79d3b9740 1 client.15585 shutdown 2024-03-26T06:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.436+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3800135242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d5360550 msgr2=0x5601d53c6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.436+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d5360550 0x5601d53c6390 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fa7840cbb40 tx=0x7fa784083f90 comp rx=0 tx=0).stop 2024-03-26T06:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.437+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3800135242 shutdown_connections 2024-03-26T06:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.437+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5601d53c8230 0x5601d53ca650 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.437+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5601d5360550 0x5601d53c6390 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.437+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5601d535d6c0 0x5601d5363e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:28.440 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.437+0000 7fa79d3b9740 1 --2- 172.21.15.38:0/3800135242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5601d531e5b0 0x5601d5363940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:28.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.437+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3800135242 >> 172.21.15.38:0/3800135242 conn(0x5601d534d190 msgr2=0x5601d53582a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:28.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.437+0000 7fa7737fe640 0 client.15585 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:28.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.437+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3800135242 shutdown_connections 2024-03-26T06:40:28.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.437+0000 7fa79d3b9740 1 -- 172.21.15.38:0/3800135242 wait complete. 2024-03-26T06:40:28.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.437+0000 7fa79d3b9740 20 client.15585 trim_cache size 0 max 16384 2024-03-26T06:40:28.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:40:28.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:40:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:40:28.442 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:40:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.438+0000 7fa79a7d4640 1 -- 172.21.15.38:0/878934729 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa77c0a2a10 msgr2=0x7fa77c0a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.438+0000 7fa79a7d4640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa77c0a2a10 0x7fa77c0a4ed0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x5601d5357a70 tx=0x7fa78c03f040 comp rx=0 tx=0).stop 2024-03-26T06:40:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.438+0000 7fa79a7d4640 1 -- 172.21.15.38:0/878934729 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa79411aad0 msgr2=0x7fa794116850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.438+0000 7fa79a7d4640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa79411aad0 0x7fa794116850 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa78400b3c0 tx=0x7fa78400b880 comp rx=0 tx=0).stop 2024-03-26T06:40:28.443 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.438+0000 7fa79a7d4640 1 -- 172.21.15.38:0/878934729 shutdown_connections 2024-03-26T06:40:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.439+0000 7fa79a7d4640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa77c0a2a10 0x7fa77c0a4ed0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.439+0000 7fa79a7d4640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa79411aad0 0x7fa794116850 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.439+0000 7fa79a7d4640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa79410fea0 0x7fa794116310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.439+0000 7fa79a7d4640 1 --2- 172.21.15.38:0/878934729 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa79410fa30 0x7fa79411a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.439+0000 7fa79a7d4640 1 -- 172.21.15.38:0/878934729 >> 172.21.15.38:0/878934729 conn(0x7fa79406c390 msgr2=0x7fa79411b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.439+0000 7fa79a7d4640 1 -- 172.21.15.38:0/878934729 shutdown_connections 2024-03-26T06:40:28.444 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:28.439+0000 7fa79a7d4640 1 -- 172.21.15.38:0/878934729 wait complete. 2024-03-26T06:40:28.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:28 smithi038 ceph-mon[32265]: pgmap v91: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 43 KiB/s rd, 105 MiB/s wr, 17.62k op/s 2024-03-26T06:40:28.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:28 smithi121 ceph-mon[36349]: pgmap v91: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 43 KiB/s rd, 105 MiB/s wr, 17.62k op/s 2024-03-26T06:40:28.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:28 smithi170 ceph-mon[36912]: pgmap v91: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 43 KiB/s rd, 105 MiB/s wr, 17.62k op/s 2024-03-26T06:40:30.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:30 smithi038 ceph-mon[32265]: pgmap v92: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 43 KiB/s rd, 87 MiB/s wr, 14.76k op/s 2024-03-26T06:40:30.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:30 smithi121 ceph-mon[36349]: pgmap v92: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 43 KiB/s rd, 87 MiB/s wr, 14.76k op/s 2024-03-26T06:40:30.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:30 smithi170 ceph-mon[36912]: pgmap v92: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 43 KiB/s rd, 87 MiB/s wr, 14.76k op/s 2024-03-26T06:40:32.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:32 smithi038 ceph-mon[32265]: pgmap v93: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 76 KiB/s rd, 126 MiB/s wr, 20.78k op/s 2024-03-26T06:40:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:32 smithi121 ceph-mon[36349]: pgmap v93: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 76 KiB/s rd, 126 MiB/s wr, 20.78k op/s 2024-03-26T06:40:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:32 smithi170 ceph-mon[36912]: pgmap v93: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 76 KiB/s rd, 126 MiB/s wr, 20.78k op/s 2024-03-26T06:40:34.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:34 smithi038 ceph-mon[32265]: pgmap v94: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 55 KiB/s rd, 82 MiB/s wr, 14.09k op/s 2024-03-26T06:40:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:34 smithi121 ceph-mon[36349]: pgmap v94: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 55 KiB/s rd, 82 MiB/s wr, 14.09k op/s 2024-03-26T06:40:34.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:34 smithi170 ceph-mon[36912]: pgmap v94: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 55 KiB/s rd, 82 MiB/s wr, 14.09k op/s 2024-03-26T06:40:36.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:36 smithi038 ceph-mon[32265]: pgmap v95: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1008 GiB / 1.0 TiB avail; 79 KiB/s rd, 95 MiB/s wr, 17.19k op/s 2024-03-26T06:40:36.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:36 smithi121 ceph-mon[36349]: pgmap v95: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1008 GiB / 1.0 TiB avail; 79 KiB/s rd, 95 MiB/s wr, 17.19k op/s 2024-03-26T06:40:36.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:36 smithi170 ceph-mon[36912]: pgmap v95: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1008 GiB / 1.0 TiB avail; 79 KiB/s rd, 95 MiB/s wr, 17.19k op/s 2024-03-26T06:40:38.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:38 smithi038 ceph-mon[32265]: pgmap v96: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 57 KiB/s rd, 84 MiB/s wr, 14.97k op/s 2024-03-26T06:40:38.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:38 smithi121 ceph-mon[36349]: pgmap v96: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 57 KiB/s rd, 84 MiB/s wr, 14.97k op/s 2024-03-26T06:40:38.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:38 smithi170 ceph-mon[36912]: pgmap v96: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 57 KiB/s rd, 84 MiB/s wr, 14.97k op/s 2024-03-26T06:40:40.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:40 smithi121 ceph-mon[36349]: pgmap v97: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 57 KiB/s rd, 77 MiB/s wr, 13.75k op/s 2024-03-26T06:40:40.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:40 smithi170 ceph-mon[36912]: pgmap v97: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 57 KiB/s rd, 77 MiB/s wr, 13.75k op/s 2024-03-26T06:40:40.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:40 smithi038 ceph-mon[32265]: pgmap v97: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 57 KiB/s rd, 77 MiB/s wr, 13.75k op/s 2024-03-26T06:40:41.626 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.623+0000 7ff693fff640 1 -- 172.21.15.38:0/2239106852 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698034e50 con 0x7ff69c10d3d0 2024-03-26T06:40:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.624+0000 7ff6a2547640 1 -- 172.21.15.38:0/2239106852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff69c10d3d0 msgr2=0x7ff69c10d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:41.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.624+0000 7ff6a2547640 1 --2- 172.21.15.38:0/2239106852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff69c10d3d0 0x7ff69c10d850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff698009860 tx=0x7ff69802bb90 comp rx=0 tx=0).stop 2024-03-26T06:40:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.625+0000 7ff6a2547640 1 -- 172.21.15.38:0/2239106852 shutdown_connections 2024-03-26T06:40:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.625+0000 7ff6a2547640 1 --2- 172.21.15.38:0/2239106852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff69c10def0 0x7ff69c11cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.625+0000 7ff6a2547640 1 --2- 172.21.15.38:0/2239106852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff69c10d3d0 0x7ff69c10d850 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff698009860 tx=0x7ff69802bb90 comp rx=0 tx=0).stop 2024-03-26T06:40:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.625+0000 7ff6a2547640 1 --2- 172.21.15.38:0/2239106852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff69c10f870 0x7ff69c10fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.625+0000 7ff6a2547640 1 -- 172.21.15.38:0/2239106852 >> 172.21.15.38:0/2239106852 conn(0x7ff69c06c200 msgr2=0x7ff69c06c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:41.633 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.629+0000 7ff6a2547640 1 -- 172.21.15.38:0/2239106852 shutdown_connections 2024-03-26T06:40:41.637 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.633+0000 7ff6a2547640 1 -- 172.21.15.38:0/2239106852 wait complete. 2024-03-26T06:40:41.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.634+0000 7ff6a2547640 1 Processor -- start 2024-03-26T06:40:41.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.634+0000 7ff6a2547640 1 -- start start 2024-03-26T06:40:41.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.634+0000 7ff6a2547640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff69c10def0 0x7ff69c11c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.634+0000 7ff6a2547640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff69c10f870 0x7ff69c115410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.638 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.634+0000 7ff6a2547640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff69c115950 0x7ff69c115e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.634+0000 7ff6a2547640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c1202d0 con 0x7ff69c10def0 2024-03-26T06:40:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.634+0000 7ff6a2547640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c120150 con 0x7ff69c115950 2024-03-26T06:40:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.634+0000 7ff6a2547640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c120450 con 0x7ff69c10f870 2024-03-26T06:40:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.634+0000 7ff6a1545640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff69c10def0 0x7ff69c11c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.639 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.635+0000 7ff6a1d46640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff69c115950 0x7ff69c115e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:41 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:40:41.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.635+0000 7ff6a1545640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff69c10def0 0x7ff69c11c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:50694/0 (socket says 172.21.15.38:50694) 2024-03-26T06:40:41.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.635+0000 7ff6a1545640 1 -- 172.21.15.38:0/97343987 learned_addr learned my addr 172.21.15.38:0/97343987 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:41.650 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.635+0000 7ff6a1d46640 1 -- 172.21.15.38:0/97343987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff69c10f870 msgr2=0x7ff69c115410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.635+0000 7ff6a1d46640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff69c10f870 0x7ff69c115410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.635+0000 7ff6a1d46640 1 -- 172.21.15.38:0/97343987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff69c10def0 msgr2=0x7ff69c11c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.635+0000 7ff6a1d46640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff69c10def0 0x7ff69c11c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.635+0000 7ff6a1d46640 1 -- 172.21.15.38:0/97343987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff69c11a050 con 0x7ff69c115950 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.635+0000 7ff6a1d46640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff69c115950 0x7ff69c115e00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff69400ef00 tx=0x7ff69400c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.636+0000 7ff6927fc640 1 -- 172.21.15.38:0/97343987 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff694004b20 con 0x7ff69c115950 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.636+0000 7ff6a2547640 1 -- 172.21.15.38:0/97343987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff69c11a340 con 0x7ff69c115950 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.636+0000 7ff6927fc640 1 -- 172.21.15.38:0/97343987 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff694003e50 con 0x7ff69c115950 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.637+0000 7ff6927fc640 1 -- 172.21.15.38:0/97343987 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69401a020 con 0x7ff69c115950 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.637+0000 7ff6a2547640 1 -- 172.21.15.38:0/97343987 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69c11a7d0 con 0x7ff69c115950 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.641+0000 7ff6a6383740 1 Processor -- start 2024-03-26T06:40:41.651 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.641+0000 7ff6a6383740 1 -- start start 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.641+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b18ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.641+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 0x55ff1b190940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.641+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 0x55ff1b19be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.641+0000 7ff6a6383740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff1b0f45f0 con 0x55ff1b199a40 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.641+0000 7ff6a6383740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff1b09d930 con 0x55ff1b1904c0 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.641+0000 7ff6a6383740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff1b0f0690 con 0x55ff1b191230 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.643+0000 7ff6927fc640 1 -- 172.21.15.38:0/97343987 <== mon.2 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff69401d020 con 0x7ff69c115950 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.644+0000 7ff6927fc640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6880a2a30 0x7ff6880a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.644+0000 7ff6927fc640 1 -- 172.21.15.38:0/97343987 <== mon.2 v2:172.21.15.121:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7ff6940cecc0 con 0x7ff69c115950 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.645+0000 7ff6a1545640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 0x55ff1b190940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.645+0000 7ff6a1545640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 0x55ff1b190940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:55138/0 (socket says 172.21.15.38:55138) 2024-03-26T06:40:41.652 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.645+0000 7ff6a1545640 1 -- 172.21.15.38:0/4277879305 learned_addr learned my addr 172.21.15.38:0/4277879305 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.645+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/4277879305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b18ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.645+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/4277879305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 0x55ff1b19be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.645+0000 7ff6a0d44640 1 -- 172.21.15.38:0/4277879305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 msgr2=0x55ff1b190940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.645+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/4277879305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 0x55ff1b190940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.645+0000 7ff6a0d44640 1 -- 172.21.15.38:0/4277879305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 msgr2=0x55ff1b19be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.645+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/4277879305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 0x55ff1b19be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.645+0000 7ff6a0d44640 1 -- 172.21.15.38:0/4277879305 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff1ae09210 con 0x55ff1b191230 2024-03-26T06:40:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.646+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/4277879305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b18ff80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff698008000 tx=0x7ff698005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c29130ae89fe9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.647+0000 7ff686ffd640 1 -- 172.21.15.38:0/4277879305 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698039070 con 0x55ff1b191230 2024-03-26T06:40:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.647+0000 7ff686ffd640 1 -- 172.21.15.38:0/4277879305 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff69803a040 con 0x55ff1b191230 2024-03-26T06:40:41.653 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.647+0000 7ff686ffd640 1 -- 172.21.15.38:0/4277879305 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff69802cac0 con 0x55ff1b191230 2024-03-26T06:40:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.649+0000 7ff6a6383740 1 -- 172.21.15.38:0/4277879305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 msgr2=0x55ff1b18ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.649+0000 7ff6a6383740 1 --2- 172.21.15.38:0/4277879305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b18ff80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff698008000 tx=0x7ff698005a90 comp rx=0 tx=0).stop 2024-03-26T06:40:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.650+0000 7ff6a1d46640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6880a2a30 0x7ff6880a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.654 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.650+0000 7ff6a1d46640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6880a2a30 0x7ff6880a4ef0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff6940cc400 tx=0x7ff6940bb040 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:41.655 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.651+0000 7ff6a6383740 1 -- 172.21.15.38:0/4277879305 shutdown_connections 2024-03-26T06:40:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.651+0000 7ff6a6383740 1 --2- 172.21.15.38:0/4277879305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 0x55ff1b19be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.651+0000 7ff6a6383740 1 --2- 172.21.15.38:0/4277879305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 0x55ff1b190940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.651+0000 7ff6a6383740 1 --2- 172.21.15.38:0/4277879305 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b18ff80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.651+0000 7ff6a6383740 1 -- 172.21.15.38:0/4277879305 >> 172.21.15.38:0/4277879305 conn(0x55ff1b1575b0 msgr2=0x55ff1b18b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.651+0000 7ff6a6383740 1 -- 172.21.15.38:0/4277879305 shutdown_connections 2024-03-26T06:40:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.651+0000 7ff6a6383740 1 -- 172.21.15.38:0/4277879305 wait complete. 2024-03-26T06:40:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.651+0000 7ff6a6383740 1 -- start start 2024-03-26T06:40:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.651+0000 7ff6a6383740 1 Processor -- start 2024-03-26T06:40:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.652+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b1904c0 0x55ff1b1968f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.652+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b191230 0x55ff1b196e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.657 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.652+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b199a40 0x55ff1b197370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.658 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.652+0000 7ff6a6383740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff1b19c620 con 0x55ff1b1904c0 2024-03-26T06:40:41.658 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.652+0000 7ff6a6383740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff1b0b0d90 con 0x55ff1b191230 2024-03-26T06:40:41.658 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.652+0000 7ff6a6383740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff1b19c7a0 con 0x55ff1b199a40 2024-03-26T06:40:41.658 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.652+0000 7ff6a6383740 10 client.0 _ll_register_callbacks cb 0x55ff1ae08a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:40:41.658 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.652+0000 7ff6a0d44640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b199a40 0x55ff1b197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.658 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.652+0000 7ff6a0d44640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b199a40 0x55ff1b197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:39508/0 (socket says 172.21.15.38:39508) 2024-03-26T06:40:41.658 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.652+0000 7ff6a0d44640 1 -- 172.21.15.38:0/1119155186 learned_addr learned my addr 172.21.15.38:0/1119155186 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:41.658 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.653+0000 7ff6a0d44640 1 -- 172.21.15.38:0/1119155186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b191230 msgr2=0x55ff1b196e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-26T06:40:41.660 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.653+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/1119155186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b191230 0x55ff1b196e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.660 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.653+0000 7ff6a0d44640 1 -- 172.21.15.38:0/1119155186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b1904c0 msgr2=0x55ff1b1968f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:41.660 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.653+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/1119155186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b1904c0 0x55ff1b1968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.660 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.653+0000 7ff6a0d44640 1 -- 172.21.15.38:0/1119155186 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff1aed5620 con 0x55ff1b199a40 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.653+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/1119155186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b199a40 0x55ff1b197370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff698034820 tx=0x7ff698034960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1d103d57b64b383 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.653+0000 7ff684ff9640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.654+0000 7ff684ff9640 1 -- 172.21.15.38:0/1119155186 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698051060 con 0x55ff1b199a40 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.654+0000 7ff684ff9640 1 -- 172.21.15.38:0/1119155186 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7ff69802ca50 con 0x55ff1b199a40 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.654+0000 7ff6a6383740 1 -- 172.21.15.38:0/1119155186 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff69803a040 con 0x55ff1b199a40 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.654+0000 7ff684ff9640 1 -- 172.21.15.38:0/1119155186 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff698039040 con 0x55ff1b199a40 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.654+0000 7ff6a6383740 10 client.25114 fetch_fsmap learned FSMap version 27 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.654+0000 7ff6a6383740 1 -- 172.21.15.38:0/1119155186 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ff1b19c620 con 0x55ff1b199a40 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.654+0000 7ff684ff9640 1 -- 172.21.15.38:0/1119155186 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff698005510 con 0x55ff1b199a40 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.655+0000 7ff684ff9640 1 -- 172.21.15.38:0/1119155186 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7ff6980049f0 con 0x55ff1b199a40 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.655+0000 7ff6a6383740 10 client.25114 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.655+0000 7ff6a6383740 10 client.25114 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.655+0000 7ff6a6383740 20 client.25114 populate_metadata read hostname 'smithi038' 2024-03-26T06:40:41.661 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.655+0000 7ff6a6383740 1 --2- 172.21.15.38:0/1119155186 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ff1b1cb920 0x55ff1b1cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.655+0000 7ff6a6383740 4 client.25114 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:40:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.655+0000 7ff6a6383740 1 -- 172.21.15.38:0/1119155186 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff69803a040 con 0x55ff1b1cb920 2024-03-26T06:40:41.662 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.659+0000 7ff6a1545640 1 --2- 172.21.15.38:0/1119155186 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ff1b1cb920 0x55ff1b1cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.663+0000 7ff6a1545640 1 --2- 172.21.15.38:0/1119155186 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ff1b1cb920 0x55ff1b1cdd10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47c46c6dc244b05d server_cookie=ebeb656da6f8aed in_seq=0 out_seq=0 2024-03-26T06:40:41.665 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.663+0000 7ff684ff9640 10 client.25114 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.664+0000 7ff684ff9640 1 -- 172.21.15.38:0/1119155186 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7ff69803a040 con 0x55ff1b1cb920 2024-03-26T06:40:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.664+0000 7ff684ff9640 10 client.25114 handle_command_reply: tid=0 2024-03-26T06:40:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.664+0000 7ff6a6383740 1 client.25114 shutdown 2024-03-26T06:40:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.664+0000 7ff6a6383740 1 -- 172.21.15.38:0/1119155186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b199a40 msgr2=0x55ff1b197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:41.667 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.664+0000 7ff6a6383740 1 --2- 172.21.15.38:0/1119155186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b199a40 0x55ff1b197370 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff698034820 tx=0x7ff698034960 comp rx=0 tx=0).stop 2024-03-26T06:40:41.667 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:41 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:40:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.665+0000 7ff6a6383740 1 -- 172.21.15.38:0/1119155186 shutdown_connections 2024-03-26T06:40:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.665+0000 7ff6a6383740 1 --2- 172.21.15.38:0/1119155186 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ff1b1cb920 0x55ff1b1cdd10 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.665+0000 7ff6a6383740 1 --2- 172.21.15.38:0/1119155186 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b199a40 0x55ff1b197370 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.665+0000 7ff6a6383740 1 --2- 172.21.15.38:0/1119155186 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b191230 0x55ff1b196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.665+0000 7ff6a6383740 1 --2- 172.21.15.38:0/1119155186 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b1904c0 0x55ff1b1968f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.665+0000 7ff6a6383740 1 -- 172.21.15.38:0/1119155186 >> 172.21.15.38:0/1119155186 conn(0x55ff1b1575b0 msgr2=0x55ff1b18cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.665+0000 7ff684ff9640 0 client.25114 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.666+0000 7ff6a6383740 1 -- 172.21.15.38:0/1119155186 shutdown_connections 2024-03-26T06:40:41.669 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.666+0000 7ff6a6383740 1 -- 172.21.15.38:0/1119155186 wait complete. 2024-03-26T06:40:41.670 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.666+0000 7ff6a6383740 20 client.25114 trim_cache size 0 max 16384 2024-03-26T06:40:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.704+0000 7ff6a6383740 1 Processor -- start 2024-03-26T06:40:41.707 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.704+0000 7ff6a6383740 1 -- start start 2024-03-26T06:40:41.708 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.705+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 0x55ff1b199020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.708 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.705+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b193b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.705+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 0x55ff1b1940d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.705+0000 7ff6a0d44640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 0x55ff1b199020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.705+0000 7ff6a0d44640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 0x55ff1b199020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:55146/0 (socket says 172.21.15.38:55146) 2024-03-26T06:40:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.705+0000 7ff6a1545640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b193b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.705+0000 7ff6a1545640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b193b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:39512/0 (socket says 172.21.15.38:39512) 2024-03-26T06:40:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.705+0000 7ff6a0d44640 1 -- 172.21.15.38:0/2321221851 learned_addr learned my addr 172.21.15.38:0/2321221851 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.706+0000 7ff6a6383740 1 -- 172.21.15.38:0/2321221851 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c1b5dd0 con 0x55ff1b199a40 2024-03-26T06:40:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.706+0000 7ff6a6383740 1 -- 172.21.15.38:0/2321221851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c1b3950 con 0x55ff1b1904c0 2024-03-26T06:40:41.709 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.706+0000 7ff6a6383740 1 -- 172.21.15.38:0/2321221851 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c04f8a0 con 0x55ff1b191230 2024-03-26T06:40:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.710+0000 7ff6a0d44640 1 -- 172.21.15.38:0/2321221851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 msgr2=0x55ff1b193b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:41.712 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.710+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/2321221851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b193b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.710+0000 7ff6a0d44640 1 -- 172.21.15.38:0/2321221851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 msgr2=0x55ff1b1940d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.710+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/2321221851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 0x55ff1b1940d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.710+0000 7ff6a0d44640 1 -- 172.21.15.38:0/2321221851 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff1b18f790 con 0x55ff1b1904c0 2024-03-26T06:40:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.710+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/2321221851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 0x55ff1b199020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff698037040 tx=0x7ff69802c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10222d9ea7399d84 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.710+0000 7ff6867fc640 1 -- 172.21.15.38:0/2321221851 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698051070 con 0x55ff1b1904c0 2024-03-26T06:40:41.713 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.711+0000 7ff6867fc640 1 -- 172.21.15.38:0/2321221851 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff698005850 con 0x55ff1b1904c0 2024-03-26T06:40:41.714 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.712+0000 7ff6a6383740 1 -- 172.21.15.38:0/2321221851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 msgr2=0x55ff1b199020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.712+0000 7ff6a6383740 1 --2- 172.21.15.38:0/2321221851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 0x55ff1b199020 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff698037040 tx=0x7ff69802c470 comp rx=0 tx=0).stop 2024-03-26T06:40:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.712+0000 7ff6a6383740 1 -- 172.21.15.38:0/2321221851 shutdown_connections 2024-03-26T06:40:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.712+0000 7ff6a6383740 1 --2- 172.21.15.38:0/2321221851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 0x55ff1b1940d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.712+0000 7ff6a6383740 1 --2- 172.21.15.38:0/2321221851 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b193b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.712+0000 7ff6a6383740 1 --2- 172.21.15.38:0/2321221851 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b1904c0 0x55ff1b199020 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.712+0000 7ff6a6383740 1 -- 172.21.15.38:0/2321221851 >> 172.21.15.38:0/2321221851 conn(0x55ff1b1575b0 msgr2=0x55ff1b185e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.712+0000 7ff6a6383740 1 -- 172.21.15.38:0/2321221851 shutdown_connections 2024-03-26T06:40:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.712+0000 7ff6a6383740 1 -- 172.21.15.38:0/2321221851 wait complete. 2024-03-26T06:40:41.715 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.712+0000 7ff6a6383740 1 -- start start 2024-03-26T06:40:41.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.713+0000 7ff6a6383740 1 Processor -- start 2024-03-26T06:40:41.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.713+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b19eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.713+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 0x55ff1b19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.713+0000 7ff6a6383740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b21aba0 0x55ff1b21b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.716 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.713+0000 7ff6a6383740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c1b1ff0 con 0x55ff1b199a40 2024-03-26T06:40:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.713+0000 7ff6a6383740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c1b32f0 con 0x55ff1b21aba0 2024-03-26T06:40:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.713+0000 7ff6a6383740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69c1b2630 con 0x55ff1b191230 2024-03-26T06:40:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.713+0000 7ff6a6383740 10 client.0 _ll_register_callbacks cb 0x7ff698037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:40:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.714+0000 7ff6a0d44640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b19eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.714+0000 7ff6a0d44640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b19eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:39520/0 (socket says 172.21.15.38:39520) 2024-03-26T06:40:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.714+0000 7ff6a0d44640 1 -- 172.21.15.38:0/118011077 learned_addr learned my addr 172.21.15.38:0/118011077 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.714+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b21aba0 0x55ff1b21b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.714+0000 7ff6a0d44640 1 -- 172.21.15.38:0/118011077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b21aba0 msgr2=0x55ff1b21b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:41.717 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.714+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b21aba0 0x55ff1b21b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.714+0000 7ff6a0d44640 1 -- 172.21.15.38:0/118011077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 msgr2=0x55ff1b19f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.714+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 0x55ff1b19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:41.718 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.714+0000 7ff6a0d44640 1 -- 172.21.15.38:0/118011077 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff1b21b5e0 con 0x55ff1b191230 2024-03-26T06:40:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.719+0000 7ff6a0d44640 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b19eea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff6980094d0 tx=0x7ff69802c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=403b6cf879f1b02b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:41.721 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.719+0000 7ff684ff9640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-26T06:40:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.720+0000 7ff684ff9640 1 -- 172.21.15.38:0/118011077 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff698051070 con 0x55ff1b191230 2024-03-26T06:40:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.720+0000 7ff6a6383740 1 -- 172.21.15.38:0/118011077 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ff1b101b30 con 0x55ff1b191230 2024-03-26T06:40:41.723 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.720+0000 7ff684ff9640 1 -- 172.21.15.38:0/118011077 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7ff6980048d0 con 0x55ff1b191230 2024-03-26T06:40:41.724 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.721+0000 7ff684ff9640 1 -- 172.21.15.38:0/118011077 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff69804c450 con 0x55ff1b191230 2024-03-26T06:40:41.724 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.721+0000 7ff6a6383740 10 client.25120 fetch_fsmap learned FSMap version 27 2024-03-26T06:40:41.724 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.721+0000 7ff6a6383740 1 -- 172.21.15.38:0/118011077 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ff1b222a80 con 0x55ff1b191230 2024-03-26T06:40:41.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.722+0000 7ff684ff9640 1 -- 172.21.15.38:0/118011077 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ff1b101b30 con 0x55ff1b191230 2024-03-26T06:40:41.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.722+0000 7ff684ff9640 1 -- 172.21.15.38:0/118011077 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7ff698056420 con 0x55ff1b191230 2024-03-26T06:40:41.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.722+0000 7ff6a6383740 10 client.25120 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:40:41.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.722+0000 7ff6a6383740 10 client.25120 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:40:41.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.722+0000 7ff6a6383740 20 client.25120 populate_metadata read hostname 'smithi038' 2024-03-26T06:40:41.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.722+0000 7ff6a6383740 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ff1b223150 0x55ff1b225570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:41.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.722+0000 7ff6a6383740 4 client.25120 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:40:41.725 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.722+0000 7ff6a6383740 1 -- 172.21.15.38:0/118011077 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ff1b0fbe80 con 0x55ff1b223150 2024-03-26T06:40:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.725+0000 7ff6a1545640 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ff1b223150 0x55ff1b225570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.726+0000 7ff6a1545640 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ff1b223150 0x55ff1b225570 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22966004f29bfde server_cookie=bdcbafed17ea3307 in_seq=0 out_seq=0 2024-03-26T06:40:41.731 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:41.726+0000 7ff684ff9640 10 client.25120 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:41.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:41 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:40:42.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:42 smithi121 ceph-mon[36349]: pgmap v98: 97 pgs: 97 active+clean; 13 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 70 KiB/s rd, 116 MiB/s wr, 20.83k op/s 2024-03-26T06:40:42.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:42 smithi170 ceph-mon[36912]: pgmap v98: 97 pgs: 97 active+clean; 13 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 70 KiB/s rd, 116 MiB/s wr, 20.83k op/s 2024-03-26T06:40:42.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:42 smithi038 ceph-mon[32265]: pgmap v98: 97 pgs: 97 active+clean; 13 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 70 KiB/s rd, 116 MiB/s wr, 20.83k op/s 2024-03-26T06:40:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:43 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:40:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:43 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:40:43.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:43 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:40:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:43 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:40:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:43 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:40:43.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:43 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:40:43.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:43 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:40:43.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:43 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:40:43.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:43 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:40:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.026+0000 7ff684ff9640 1 -- 172.21.15.38:0/118011077 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ff1b0fbe80 con 0x55ff1b223150 2024-03-26T06:40:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.026+0000 7ff684ff9640 10 client.25120 handle_command_reply: tid=0 2024-03-26T06:40:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.026+0000 7ff6a6383740 1 client.25120 shutdown 2024-03-26T06:40:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.026+0000 7ff6a6383740 1 -- 172.21.15.38:0/118011077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 msgr2=0x55ff1b19eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.026+0000 7ff6a6383740 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b19eea0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff6980094d0 tx=0x7ff69802c7d0 comp rx=0 tx=0).stop 2024-03-26T06:40:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.027+0000 7ff6a6383740 1 -- 172.21.15.38:0/118011077 shutdown_connections 2024-03-26T06:40:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.027+0000 7ff6a6383740 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55ff1b223150 0x55ff1b225570 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.027+0000 7ff6a6383740 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55ff1b21aba0 0x55ff1b21b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.027+0000 7ff6a6383740 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ff1b199a40 0x55ff1b19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.027+0000 7ff6a6383740 1 --2- 172.21.15.38:0/118011077 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55ff1b191230 0x55ff1b19eea0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.027+0000 7ff6a6383740 1 -- 172.21.15.38:0/118011077 >> 172.21.15.38:0/118011077 conn(0x55ff1b1575b0 msgr2=0x55ff1b1951e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.027+0000 7ff684ff9640 0 client.25120 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.027+0000 7ff6a6383740 1 -- 172.21.15.38:0/118011077 shutdown_connections 2024-03-26T06:40:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.027+0000 7ff6a6383740 1 -- 172.21.15.38:0/118011077 wait complete. 2024-03-26T06:40:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.027+0000 7ff6a6383740 20 client.25120 trim_cache size 0 max 16384 2024-03-26T06:40:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:40:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:40:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:40:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:40:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.029+0000 7ff687fff640 1 -- 172.21.15.38:0/97343987 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6880a2a30 msgr2=0x7ff6880a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.029+0000 7ff687fff640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6880a2a30 0x7ff6880a4ef0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff6940cc400 tx=0x7ff6940bb040 comp rx=0 tx=0).stop 2024-03-26T06:40:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.029+0000 7ff687fff640 1 -- 172.21.15.38:0/97343987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff69c115950 msgr2=0x7ff69c115e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.029+0000 7ff687fff640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff69c115950 0x7ff69c115e00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff69400ef00 tx=0x7ff69400c740 comp rx=0 tx=0).stop 2024-03-26T06:40:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.029+0000 7ff687fff640 1 -- 172.21.15.38:0/97343987 shutdown_connections 2024-03-26T06:40:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.029+0000 7ff687fff640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6880a2a30 0x7ff6880a4ef0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.029+0000 7ff687fff640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff69c115950 0x7ff69c115e00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.029+0000 7ff687fff640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff69c10f870 0x7ff69c115410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.030+0000 7ff687fff640 1 --2- 172.21.15.38:0/97343987 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff69c10def0 0x7ff69c11c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.030+0000 7ff687fff640 1 -- 172.21.15.38:0/97343987 >> 172.21.15.38:0/97343987 conn(0x7ff69c06c200 msgr2=0x7ff69c070710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.030+0000 7ff687fff640 1 -- 172.21.15.38:0/97343987 shutdown_connections 2024-03-26T06:40:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:44.030+0000 7ff687fff640 1 -- 172.21.15.38:0/97343987 wait complete. 2024-03-26T06:40:44.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:44 smithi038 ceph-mon[32265]: pgmap v99: 97 pgs: 97 active+clean; 13 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 37 KiB/s rd, 85 MiB/s wr, 16.08k op/s 2024-03-26T06:40:44.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:44 smithi121 ceph-mon[36349]: pgmap v99: 97 pgs: 97 active+clean; 13 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 37 KiB/s rd, 85 MiB/s wr, 16.08k op/s 2024-03-26T06:40:44.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:44 smithi170 ceph-mon[36912]: pgmap v99: 97 pgs: 97 active+clean; 13 GiB data, 66 GiB used, 1007 GiB / 1.0 TiB avail; 37 KiB/s rd, 85 MiB/s wr, 16.08k op/s 2024-03-26T06:40:46.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:46 smithi038 ceph-mon[32265]: pgmap v100: 97 pgs: 97 active+clean; 13 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 74 KiB/s rd, 91 MiB/s wr, 17.25k op/s 2024-03-26T06:40:46.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:46 smithi121 ceph-mon[36349]: pgmap v100: 97 pgs: 97 active+clean; 13 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 74 KiB/s rd, 91 MiB/s wr, 17.25k op/s 2024-03-26T06:40:46.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:46 smithi170 ceph-mon[36912]: pgmap v100: 97 pgs: 97 active+clean; 13 GiB data, 67 GiB used, 1006 GiB / 1.0 TiB avail; 74 KiB/s rd, 91 MiB/s wr, 17.25k op/s 2024-03-26T06:40:48.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:48 smithi038 ceph-mon[32265]: pgmap v101: 97 pgs: 97 active+clean; 14 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 50 KiB/s rd, 94 MiB/s wr, 17.53k op/s 2024-03-26T06:40:48.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:48 smithi121 ceph-mon[36349]: pgmap v101: 97 pgs: 97 active+clean; 14 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 50 KiB/s rd, 94 MiB/s wr, 17.53k op/s 2024-03-26T06:40:48.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:48 smithi170 ceph-mon[36912]: pgmap v101: 97 pgs: 97 active+clean; 14 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 50 KiB/s rd, 94 MiB/s wr, 17.53k op/s 2024-03-26T06:40:50.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:50 smithi038 ceph-mon[32265]: pgmap v102: 97 pgs: 97 active+clean; 14 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 50 KiB/s rd, 93 MiB/s wr, 17.05k op/s 2024-03-26T06:40:50.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:50 smithi121 ceph-mon[36349]: pgmap v102: 97 pgs: 97 active+clean; 14 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 50 KiB/s rd, 93 MiB/s wr, 17.05k op/s 2024-03-26T06:40:50.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:50 smithi170 ceph-mon[36912]: pgmap v102: 97 pgs: 97 active+clean; 14 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 50 KiB/s rd, 93 MiB/s wr, 17.05k op/s 2024-03-26T06:40:52.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:52 smithi038 ceph-mon[32265]: pgmap v103: 97 pgs: 97 active+clean; 14 GiB data, 69 GiB used, 1003 GiB / 1.0 TiB avail; 97 KiB/s rd, 128 MiB/s wr, 23.19k op/s 2024-03-26T06:40:52.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:52 smithi121 ceph-mon[36349]: pgmap v103: 97 pgs: 97 active+clean; 14 GiB data, 69 GiB used, 1003 GiB / 1.0 TiB avail; 97 KiB/s rd, 128 MiB/s wr, 23.19k op/s 2024-03-26T06:40:52.917 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:52 smithi170 ceph-mon[36912]: pgmap v103: 97 pgs: 97 active+clean; 14 GiB data, 69 GiB used, 1003 GiB / 1.0 TiB avail; 97 KiB/s rd, 128 MiB/s wr, 23.19k op/s 2024-03-26T06:40:54.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:54 smithi038 ceph-mon[32265]: pgmap v104: 97 pgs: 97 active+clean; 14 GiB data, 69 GiB used, 1003 GiB / 1.0 TiB avail; 84 KiB/s rd, 90 MiB/s wr, 16.11k op/s 2024-03-26T06:40:54.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:54 smithi121 ceph-mon[36349]: pgmap v104: 97 pgs: 97 active+clean; 14 GiB data, 69 GiB used, 1003 GiB / 1.0 TiB avail; 84 KiB/s rd, 90 MiB/s wr, 16.11k op/s 2024-03-26T06:40:54.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:54 smithi170 ceph-mon[36912]: pgmap v104: 97 pgs: 97 active+clean; 14 GiB data, 69 GiB used, 1003 GiB / 1.0 TiB avail; 84 KiB/s rd, 90 MiB/s wr, 16.11k op/s 2024-03-26T06:40:56.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:56 smithi038 ceph-mon[32265]: pgmap v105: 97 pgs: 97 active+clean; 14 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 151 KiB/s rd, 99 MiB/s wr, 17.84k op/s 2024-03-26T06:40:56.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:56 smithi121 ceph-mon[36349]: pgmap v105: 97 pgs: 97 active+clean; 14 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 151 KiB/s rd, 99 MiB/s wr, 17.84k op/s 2024-03-26T06:40:56.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:56 smithi170 ceph-mon[36912]: pgmap v105: 97 pgs: 97 active+clean; 14 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 151 KiB/s rd, 99 MiB/s wr, 17.84k op/s 2024-03-26T06:40:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.211+0000 7fc40af84640 1 -- 172.21.15.38:0/2772803583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3fc004c80 msgr2=0x7fc3fc005100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:57.214 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.211+0000 7fc40af84640 1 --2- 172.21.15.38:0/2772803583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3fc004c80 0x7fc3fc005100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc40001a910 tx=0x7fc40003cc90 comp rx=0 tx=0).stop 2024-03-26T06:40:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.220+0000 7fc40af84640 1 -- 172.21.15.38:0/2772803583 shutdown_connections 2024-03-26T06:40:57.223 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.220+0000 7fc40af84640 1 --2- 172.21.15.38:0/2772803583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3fc0986e0 0x7fc3fc098b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.220+0000 7fc40af84640 1 --2- 172.21.15.38:0/2772803583 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3fc004c80 0x7fc3fc005100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.220+0000 7fc40af84640 1 --2- 172.21.15.38:0/2772803583 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3fc09acf0 0x7fc3fc004740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.220+0000 7fc40af84640 1 -- 172.21.15.38:0/2772803583 >> 172.21.15.38:0/2772803583 conn(0x7fc3fc00a740 msgr2=0x7fc3fc00ab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:57.224 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.222+0000 7fc40af84640 1 -- 172.21.15.38:0/2772803583 shutdown_connections 2024-03-26T06:40:57.227 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.222+0000 7fc40af84640 1 -- 172.21.15.38:0/2772803583 wait complete. 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.223+0000 7fc40af84640 1 Processor -- start 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.223+0000 7fc40af84640 1 -- start start 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.223+0000 7fc40af84640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3fc004c80 0x7fc3fc0a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.223+0000 7fc40af84640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3fc0986e0 0x7fc3fc0a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc40af84640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3fc09acf0 0x7fc3fc13fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc40af84640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc09db90 con 0x7fc3fc0986e0 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc40af84640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc09da10 con 0x7fc3fc004c80 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc40af84640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc09dd10 con 0x7fc3fc09acf0 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc40a783640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3fc09acf0 0x7fc3fc13fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc409f82640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3fc004c80 0x7fc3fc0a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc409f82640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3fc004c80 0x7fc3fc0a54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:44424/0 (socket says 172.21.15.38:44424) 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc40a783640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3fc09acf0 0x7fc3fc13fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:39768/0 (socket says 172.21.15.38:39768) 2024-03-26T06:40:57.228 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc409f82640 1 -- 172.21.15.38:0/868869769 learned_addr learned my addr 172.21.15.38:0/868869769 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc40a783640 1 -- 172.21.15.38:0/868869769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3fc004c80 msgr2=0x7fc3fc0a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc40a783640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3fc004c80 0x7fc3fc0a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc40a783640 1 -- 172.21.15.38:0/868869769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3fc0986e0 msgr2=0x7fc3fc0a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc40a783640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3fc0986e0 0x7fc3fc0a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.224+0000 7fc40a783640 1 -- 172.21.15.38:0/868869769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3fc140460 con 0x7fc3fc09acf0 2024-03-26T06:40:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.225+0000 7fc40a783640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3fc09acf0 0x7fc3fc13fba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc40404f320 tx=0x7fc4040765a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.225+0000 7fc3faffd640 1 -- 172.21.15.38:0/868869769 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404076e10 con 0x7fc3fc09acf0 2024-03-26T06:40:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.225+0000 7fc40af84640 1 -- 172.21.15.38:0/868869769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3fc140750 con 0x7fc3fc09acf0 2024-03-26T06:40:57.230 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.225+0000 7fc40af84640 1 -- 172.21.15.38:0/868869769 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3fc0b0090 con 0x7fc3fc09acf0 2024-03-26T06:40:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.229+0000 7fc40edc9740 1 Processor -- start 2024-03-26T06:40:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.229+0000 7fc40edc9740 1 -- start start 2024-03-26T06:40:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.229+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fbaf20 0x559185fb9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.229+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 0x559185fba630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.229+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fc3730 0x559185fc5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.229+0000 7fc40edc9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x559185f1e5f0 con 0x559185fc3730 2024-03-26T06:40:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.229+0000 7fc40edc9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559185ec7930 con 0x559185fba1b0 2024-03-26T06:40:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.229+0000 7fc40edc9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559185f1a690 con 0x559185fbaf20 2024-03-26T06:40:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.229+0000 7fc3faffd640 1 -- 172.21.15.38:0/868869769 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4040728e0 con 0x7fc3fc09acf0 2024-03-26T06:40:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.229+0000 7fc3faffd640 1 -- 172.21.15.38:0/868869769 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40407b960 con 0x7fc3fc09acf0 2024-03-26T06:40:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.230+0000 7fc409f82640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 0x559185fba630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.230+0000 7fc409f82640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 0x559185fba630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:44426/0 (socket says 172.21.15.38:44426) 2024-03-26T06:40:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.230+0000 7fc409f82640 1 -- 172.21.15.38:0/1330481699 learned_addr learned my addr 172.21.15.38:0/1330481699 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.230+0000 7fc409f82640 1 -- 172.21.15.38:0/1330481699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fbaf20 msgr2=0x559185fb9c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-26T06:40:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.230+0000 7fc409f82640 1 --2- 172.21.15.38:0/1330481699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fbaf20 0x559185fb9c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.230+0000 7fc409f82640 1 -- 172.21.15.38:0/1330481699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fc3730 msgr2=0x559185fc5b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-26T06:40:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.230+0000 7fc409f82640 1 --2- 172.21.15.38:0/1330481699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fc3730 0x559185fc5b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.233+0000 7fc3faffd640 1 -- 172.21.15.38:0/868869769 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc40407a030 con 0x7fc3fc09acf0 2024-03-26T06:40:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.233+0000 7fc3faffd640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fc3f00a2a70 0x7fc3f00a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.234+0000 7fc3faffd640 1 -- 172.21.15.38:0/868869769 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fc404132b10 con 0x7fc3fc09acf0 2024-03-26T06:40:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.234+0000 7fc409f82640 1 -- 172.21.15.38:0/1330481699 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559185c33210 con 0x559185fba1b0 2024-03-26T06:40:57.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.235+0000 7fc409f82640 1 --2- 172.21.15.38:0/1330481699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 0x559185fba630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400bda0 tx=0x7fc3f40107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c1f2b8a744bca37 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.235+0000 7fc3effff640 1 -- 172.21.15.38:0/1330481699 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f401c680 con 0x559185fba1b0 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.235+0000 7fc3effff640 1 -- 172.21.15.38:0/1330481699 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3f4011920 con 0x559185fba1b0 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.235+0000 7fc3effff640 1 -- 172.21.15.38:0/1330481699 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f401a920 con 0x559185fba1b0 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.236+0000 7fc40edc9740 1 -- 172.21.15.38:0/1330481699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 msgr2=0x559185fba630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.236+0000 7fc40edc9740 1 --2- 172.21.15.38:0/1330481699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 0x559185fba630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400bda0 tx=0x7fc3f40107a0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.237+0000 7fc40edc9740 1 -- 172.21.15.38:0/1330481699 shutdown_connections 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.237+0000 7fc40edc9740 1 --2- 172.21.15.38:0/1330481699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fc3730 0x559185fc5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.237+0000 7fc40edc9740 1 --2- 172.21.15.38:0/1330481699 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 0x559185fba630 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.237+0000 7fc40edc9740 1 --2- 172.21.15.38:0/1330481699 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fbaf20 0x559185fb9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.237+0000 7fc40edc9740 1 -- 172.21.15.38:0/1330481699 >> 172.21.15.38:0/1330481699 conn(0x559185f815b0 msgr2=0x559185fb4ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.243+0000 7fc409781640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fc3f00a2a70 0x7fc3f00a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.247 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.243+0000 7fc40edc9740 1 -- 172.21.15.38:0/1330481699 shutdown_connections 2024-03-26T06:40:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.243+0000 7fc40edc9740 1 -- 172.21.15.38:0/1330481699 wait complete. 2024-03-26T06:40:57.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.243+0000 7fc40edc9740 1 -- start start 2024-03-26T06:40:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.243+0000 7fc40edc9740 1 Processor -- start 2024-03-26T06:40:57.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.244+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 0x559185fc0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.244+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fbaf20 0x559185fc0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.244+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fc3730 0x559185fc11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.244+0000 7fc40edc9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x559185fc6460 con 0x559185fc3730 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.244+0000 7fc40edc9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x559185edad90 con 0x559185fba1b0 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.244+0000 7fc40edc9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x559185fc65e0 con 0x559185fbaf20 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.244+0000 7fc40edc9740 10 client.0 _ll_register_callbacks cb 0x559185c32a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.244+0000 7fc409781640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fbaf20 0x559185fc0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.244+0000 7fc409781640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fbaf20 0x559185fc0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:39776/0 (socket says 172.21.15.38:39776) 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.244+0000 7fc409781640 1 -- 172.21.15.38:0/2248506269 learned_addr learned my addr 172.21.15.38:0/2248506269 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.245+0000 7fc40a783640 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fc3730 0x559185fc11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.245+0000 7fc409f82640 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 0x559185fc0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.245+0000 7fc409781640 1 -- 172.21.15.38:0/2248506269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 msgr2=0x559185fc0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.245+0000 7fc409781640 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 0x559185fc0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.245+0000 7fc409781640 1 -- 172.21.15.38:0/2248506269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fc3730 msgr2=0x559185fc11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.245+0000 7fc409781640 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fc3730 0x559185fc11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.245+0000 7fc409781640 1 -- 172.21.15.38:0/2248506269 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559185cff620 con 0x559185fbaf20 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.245+0000 7fc409781640 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fbaf20 0x559185fc0c70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc400005ec0 tx=0x7fc4000040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2436f814990e81a server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.245+0000 7fc3ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.246+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/2248506269 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40003d840 con 0x559185fbaf20 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.246+0000 7fc40edc9740 1 -- 172.21.15.38:0/2248506269 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3f4011920 con 0x559185fbaf20 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.246+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/2248506269 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fc4000035f0 con 0x559185fbaf20 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.246+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/2248506269 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4000044c0 con 0x559185fbaf20 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.246+0000 7fc40edc9740 10 client.25129 fetch_fsmap learned FSMap version 27 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.246+0000 7fc40edc9740 1 -- 172.21.15.38:0/2248506269 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x559185fd2810 con 0x559185fbaf20 2024-03-26T06:40:57.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.247+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/2248506269 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3f4011920 con 0x559185fbaf20 2024-03-26T06:40:57.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.247+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/2248506269 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc400051050 con 0x559185fbaf20 2024-03-26T06:40:57.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.247+0000 7fc40edc9740 10 client.25129 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:40:57.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.247+0000 7fc40edc9740 10 client.25129 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:40:57.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.247+0000 7fc40edc9740 20 client.25129 populate_metadata read hostname 'smithi038' 2024-03-26T06:40:57.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.247+0000 7fc40edc9740 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x559185fd4200 0x559185fd6620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.247+0000 7fc40edc9740 4 client.25129 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:40:57.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.247+0000 7fc40edc9740 1 -- 172.21.15.38:0/2248506269 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559185f2bb30 con 0x559185fd4200 2024-03-26T06:40:57.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.248+0000 7fc409781640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fc3f00a2a70 0x7fc3f00a4f30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc40004c590 tx=0x7fc40004c5c0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:57.261 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.250+0000 7fc409f82640 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x559185fd4200 0x559185fd6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.251+0000 7fc409f82640 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x559185fd4200 0x559185fd6620 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37658fdb7bf9a412 server_cookie=60ea0329c344d226 in_seq=0 out_seq=0 2024-03-26T06:40:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.251+0000 7fc3ed7fa640 10 client.25129 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.252+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/2248506269 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559185f2bb30 con 0x559185fd4200 2024-03-26T06:40:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.252+0000 7fc3ed7fa640 10 client.25129 handle_command_reply: tid=0 2024-03-26T06:40:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.252+0000 7fc40edc9740 1 client.25129 shutdown 2024-03-26T06:40:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.252+0000 7fc40edc9740 1 -- 172.21.15.38:0/2248506269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fbaf20 msgr2=0x559185fc0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.252+0000 7fc40edc9740 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fbaf20 0x559185fc0c70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc400005ec0 tx=0x7fc4000040c0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.253+0000 7fc40edc9740 1 -- 172.21.15.38:0/2248506269 shutdown_connections 2024-03-26T06:40:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.253+0000 7fc40edc9740 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x559185fd4200 0x559185fd6620 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.253+0000 7fc40edc9740 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fc3730 0x559185fc11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.262 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.253+0000 7fc40edc9740 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fbaf20 0x559185fc0c70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.253+0000 7fc40edc9740 1 --2- 172.21.15.38:0/2248506269 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fba1b0 0x559185fc0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.253+0000 7fc40edc9740 1 -- 172.21.15.38:0/2248506269 >> 172.21.15.38:0/2248506269 conn(0x559185f815b0 msgr2=0x559185fb6bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.253+0000 7fc3ed7fa640 0 client.25129 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.253+0000 7fc40edc9740 1 -- 172.21.15.38:0/2248506269 shutdown_connections 2024-03-26T06:40:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.253+0000 7fc40edc9740 1 -- 172.21.15.38:0/2248506269 wait complete. 2024-03-26T06:40:57.263 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.253+0000 7fc40edc9740 20 client.25129 trim_cache size 0 max 16384 2024-03-26T06:40:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.284+0000 7fc40edc9740 1 Processor -- start 2024-03-26T06:40:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.284+0000 7fc40edc9740 1 -- start start 2024-03-26T06:40:57.290 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.284+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fba1b0 0x559185fc2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.284+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 0x559185fca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.284+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fc3730 0x559185fcae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.284+0000 7fc40edc9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e8010ab0 con 0x559185fbaf20 2024-03-26T06:40:57.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.284+0000 7fc40edc9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e800e5f0 con 0x559185fc3730 2024-03-26T06:40:57.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.284+0000 7fc409f82640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fba1b0 0x559185fc2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.291 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.285+0000 7fc40a783640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fc3730 0x559185fcae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.285+0000 7fc409781640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 0x559185fca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.285+0000 7fc409781640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 0x559185fca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41960/0 (socket says 172.21.15.38:41960) 2024-03-26T06:40:57.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.285+0000 7fc409781640 1 -- 172.21.15.38:0/572997049 learned_addr learned my addr 172.21.15.38:0/572997049 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:57.292 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.285+0000 7fc40edc9740 1 -- 172.21.15.38:0/572997049 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e800c690 con 0x559185fba1b0 2024-03-26T06:40:57.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.287+0000 7fc409f82640 1 -- 172.21.15.38:0/572997049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fc3730 msgr2=0x559185fcae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:57.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.287+0000 7fc409f82640 1 --2- 172.21.15.38:0/572997049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fc3730 0x559185fcae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.287+0000 7fc409f82640 1 -- 172.21.15.38:0/572997049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 msgr2=0x559185fca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:57.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.287+0000 7fc409f82640 1 --2- 172.21.15.38:0/572997049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 0x559185fca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.302 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.287+0000 7fc409f82640 1 -- 172.21.15.38:0/572997049 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559185faf2f0 con 0x559185fba1b0 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.287+0000 7fc409f82640 1 --2- 172.21.15.38:0/572997049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fba1b0 0x559185fc2e60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4010ca0 tx=0x7fc3f4011db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71ad59f29124d472 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.288+0000 7fc3ef7fe640 1 -- 172.21.15.38:0/572997049 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4011440 con 0x559185fba1b0 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.288+0000 7fc3ef7fe640 1 -- 172.21.15.38:0/572997049 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3f404f030 con 0x559185fba1b0 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.289+0000 7fc3ef7fe640 1 -- 172.21.15.38:0/572997049 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f40035f0 con 0x559185fba1b0 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.289+0000 7fc40edc9740 1 -- 172.21.15.38:0/572997049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fba1b0 msgr2=0x559185fc2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.289+0000 7fc40edc9740 1 --2- 172.21.15.38:0/572997049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fba1b0 0x559185fc2e60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4010ca0 tx=0x7fc3f4011db0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.290+0000 7fc40edc9740 1 -- 172.21.15.38:0/572997049 shutdown_connections 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.290+0000 7fc40edc9740 1 --2- 172.21.15.38:0/572997049 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559185fc3730 0x559185fcae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.290+0000 7fc40edc9740 1 --2- 172.21.15.38:0/572997049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 0x559185fca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.290+0000 7fc40edc9740 1 --2- 172.21.15.38:0/572997049 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fba1b0 0x559185fc2e60 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4010ca0 tx=0x7fc3f4011db0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.290+0000 7fc40edc9740 1 -- 172.21.15.38:0/572997049 >> 172.21.15.38:0/572997049 conn(0x559185f815b0 msgr2=0x559185fdafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.290+0000 7fc40edc9740 1 -- 172.21.15.38:0/572997049 shutdown_connections 2024-03-26T06:40:57.303 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.290+0000 7fc40edc9740 1 -- 172.21.15.38:0/572997049 wait complete. 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.291+0000 7fc40edc9740 1 -- start start 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.291+0000 7fc40edc9740 1 Processor -- start 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.291+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 0x559185fc6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.291+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fc3730 0x559185fc6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.291+0000 7fc40edc9740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559186027700 0x559186029af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.291+0000 7fc40edc9740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e800cc90 con 0x559185fbaf20 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.291+0000 7fc40edc9740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e800df90 con 0x559186027700 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.291+0000 7fc40edc9740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e800d2d0 con 0x559185fc3730 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.291+0000 7fc40edc9740 10 client.0 _ll_register_callbacks cb 0x7fc3f400b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.292+0000 7fc409781640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fc3730 0x559185fc6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.292+0000 7fc40a783640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559186027700 0x559186029af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.292+0000 7fc40a783640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559186027700 0x559186029af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:44464/0 (socket says 172.21.15.38:44464) 2024-03-26T06:40:57.304 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.292+0000 7fc40a783640 1 -- 172.21.15.38:0/103984670 learned_addr learned my addr 172.21.15.38:0/103984670 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.292+0000 7fc409f82640 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 0x559185fc6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.293+0000 7fc409f82640 1 -- 172.21.15.38:0/103984670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fc3730 msgr2=0x559185fc6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.293+0000 7fc409f82640 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fc3730 0x559185fc6ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.293+0000 7fc409f82640 1 -- 172.21.15.38:0/103984670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559186027700 msgr2=0x559186029af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.293+0000 7fc409f82640 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559186027700 0x559186029af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.293+0000 7fc409f82640 1 -- 172.21.15.38:0/103984670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55918602a030 con 0x559185fbaf20 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.293+0000 7fc409f82640 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 0x559185fc6ab0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x559185fb5d00 tx=0x7fc3f400b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeaaeb7b73ec0b26 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.293+0000 7fc3ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.294+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/103984670 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f4005040 con 0x559185fbaf20 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.294+0000 7fc40edc9740 1 -- 172.21.15.38:0/103984670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3f404f030 con 0x559185fbaf20 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.294+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/103984670 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fc3f4005490 con 0x559185fbaf20 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.294+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/103984670 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3f4011440 con 0x559185fbaf20 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.294+0000 7fc40edc9740 10 client.15606 fetch_fsmap learned FSMap version 27 2024-03-26T06:40:57.305 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.294+0000 7fc40edc9740 1 -- 172.21.15.38:0/103984670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55918602b2f0 con 0x559185fbaf20 2024-03-26T06:40:57.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.295+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/103984670 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3f404f030 con 0x559185fbaf20 2024-03-26T06:40:57.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.295+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/103984670 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fc3f4003c00 con 0x559185fbaf20 2024-03-26T06:40:57.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.296+0000 7fc40edc9740 10 client.15606 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:40:57.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.296+0000 7fc40edc9740 10 client.15606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:40:57.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.296+0000 7fc40edc9740 20 client.15606 populate_metadata read hostname 'smithi038' 2024-03-26T06:40:57.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.296+0000 7fc40edc9740 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55918602b8b0 0x55918602dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:40:57.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.296+0000 7fc40edc9740 4 client.15606 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:40:57.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.296+0000 7fc40edc9740 1 -- 172.21.15.38:0/103984670 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559185f25e80 con 0x55918602b8b0 2024-03-26T06:40:57.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.296+0000 7fc409781640 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55918602b8b0 0x55918602dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:40:57.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.298+0000 7fc409781640 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55918602b8b0 0x55918602dcd0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fee26dea0bb69cdf server_cookie=3e28068c72ec8d2b in_seq=0 out_seq=0 2024-03-26T06:40:57.306 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:57.298+0000 7fc3ed7fa640 10 client.15606 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.272+0000 7fc3ed7fa640 1 -- 172.21.15.38:0/103984670 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559185f25e80 con 0x55918602b8b0 2024-03-26T06:40:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.272+0000 7fc3ed7fa640 10 client.15606 handle_command_reply: tid=0 2024-03-26T06:40:58.276 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.272+0000 7fc40edc9740 1 client.15606 shutdown 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.272+0000 7fc40edc9740 1 -- 172.21.15.38:0/103984670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 msgr2=0x559185fc6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.272+0000 7fc40edc9740 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 0x559185fc6ab0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x559185fb5d00 tx=0x7fc3f400b970 comp rx=0 tx=0).stop 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.273+0000 7fc40edc9740 1 -- 172.21.15.38:0/103984670 shutdown_connections 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.273+0000 7fc40edc9740 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55918602b8b0 0x55918602dcd0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.273+0000 7fc40edc9740 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x559186027700 0x559186029af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.273+0000 7fc40edc9740 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x559185fc3730 0x559185fc6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.273+0000 7fc40edc9740 1 --2- 172.21.15.38:0/103984670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559185fbaf20 0x559185fc6ab0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.273+0000 7fc40edc9740 1 -- 172.21.15.38:0/103984670 >> 172.21.15.38:0/103984670 conn(0x559185f815b0 msgr2=0x559185fbba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.273+0000 7fc3ed7fa640 0 client.15606 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.273+0000 7fc40edc9740 1 -- 172.21.15.38:0/103984670 shutdown_connections 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.273+0000 7fc40edc9740 1 -- 172.21.15.38:0/103984670 wait complete. 2024-03-26T06:40:58.277 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.273+0000 7fc40edc9740 20 client.15606 trim_cache size 0 max 16384 2024-03-26T06:40:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:40:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:40:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:40:58.278 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.274+0000 7fc40af84640 1 -- 172.21.15.38:0/868869769 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fc3f00a2a70 msgr2=0x7fc3f00a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.274+0000 7fc40af84640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fc3f00a2a70 0x7fc3f00a4f30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc40004c590 tx=0x7fc40004c5c0 comp rx=0 tx=0).stop 2024-03-26T06:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.274+0000 7fc40af84640 1 -- 172.21.15.38:0/868869769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3fc09acf0 msgr2=0x7fc3fc13fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.274+0000 7fc40af84640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3fc09acf0 0x7fc3fc13fba0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fc40404f320 tx=0x7fc4040765a0 comp rx=0 tx=0).stop 2024-03-26T06:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.274+0000 7fc40af84640 1 -- 172.21.15.38:0/868869769 shutdown_connections 2024-03-26T06:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.274+0000 7fc40af84640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fc3f00a2a70 0x7fc3f00a4f30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.274+0000 7fc40af84640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fc3fc09acf0 0x7fc3fc13fba0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.274+0000 7fc40af84640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3fc0986e0 0x7fc3fc0a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.274+0000 7fc40af84640 1 --2- 172.21.15.38:0/868869769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3fc004c80 0x7fc3fc0a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.274+0000 7fc40af84640 1 -- 172.21.15.38:0/868869769 >> 172.21.15.38:0/868869769 conn(0x7fc3fc00a740 msgr2=0x7fc3fc092310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:40:58.280 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.275+0000 7fc40af84640 1 -- 172.21.15.38:0/868869769 shutdown_connections 2024-03-26T06:40:58.281 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:40:58.275+0000 7fc40af84640 1 -- 172.21.15.38:0/868869769 wait complete. 2024-03-26T06:40:58.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:40:58 smithi038 ceph-mon[32265]: pgmap v106: 97 pgs: 97 active+clean; 14 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 115 KiB/s rd, 126 MiB/s wr, 22.74k op/s 2024-03-26T06:40:58.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:40:58 smithi121 ceph-mon[36349]: pgmap v106: 97 pgs: 97 active+clean; 14 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 115 KiB/s rd, 126 MiB/s wr, 22.74k op/s 2024-03-26T06:40:58.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:40:58 smithi170 ceph-mon[36912]: pgmap v106: 97 pgs: 97 active+clean; 14 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 115 KiB/s rd, 126 MiB/s wr, 22.74k op/s 2024-03-26T06:41:00.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:00 smithi121 ceph-mon[36349]: pgmap v107: 97 pgs: 97 active+clean; 14 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 114 KiB/s rd, 107 MiB/s wr, 19.27k op/s 2024-03-26T06:41:00.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:00 smithi170 ceph-mon[36912]: pgmap v107: 97 pgs: 97 active+clean; 14 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 114 KiB/s rd, 107 MiB/s wr, 19.27k op/s 2024-03-26T06:41:01.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:00 smithi038 ceph-mon[32265]: pgmap v107: 97 pgs: 97 active+clean; 14 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 114 KiB/s rd, 107 MiB/s wr, 19.27k op/s 2024-03-26T06:41:02.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:02 smithi121 ceph-mon[36349]: pgmap v108: 97 pgs: 97 active+clean; 15 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 348 KiB/s rd, 147 MiB/s wr, 24.87k op/s 2024-03-26T06:41:02.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:02 smithi170 ceph-mon[36912]: pgmap v108: 97 pgs: 97 active+clean; 15 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 348 KiB/s rd, 147 MiB/s wr, 24.87k op/s 2024-03-26T06:41:03.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:02 smithi038 ceph-mon[32265]: pgmap v108: 97 pgs: 97 active+clean; 15 GiB data, 70 GiB used, 1003 GiB / 1.0 TiB avail; 348 KiB/s rd, 147 MiB/s wr, 24.87k op/s 2024-03-26T06:41:04.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:04 smithi121 ceph-mon[36349]: pgmap v109: 97 pgs: 97 active+clean; 15 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 301 KiB/s rd, 116 MiB/s wr, 19.10k op/s 2024-03-26T06:41:04.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:04 smithi170 ceph-mon[36912]: pgmap v109: 97 pgs: 97 active+clean; 15 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 301 KiB/s rd, 116 MiB/s wr, 19.10k op/s 2024-03-26T06:41:05.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:04 smithi038 ceph-mon[32265]: pgmap v109: 97 pgs: 97 active+clean; 15 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 301 KiB/s rd, 116 MiB/s wr, 19.10k op/s 2024-03-26T06:41:06.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:06 smithi121 ceph-mon[36349]: pgmap v110: 97 pgs: 97 active+clean; 15 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 308 KiB/s rd, 133 MiB/s wr, 21.34k op/s 2024-03-26T06:41:06.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:06 smithi170 ceph-mon[36912]: pgmap v110: 97 pgs: 97 active+clean; 15 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 308 KiB/s rd, 133 MiB/s wr, 21.34k op/s 2024-03-26T06:41:07.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:06 smithi038 ceph-mon[32265]: pgmap v110: 97 pgs: 97 active+clean; 15 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 308 KiB/s rd, 133 MiB/s wr, 21.34k op/s 2024-03-26T06:41:07.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:07 smithi121 ceph-mon[36349]: pgmap v111: 97 pgs: 97 active+clean; 16 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 383 KiB/s rd, 158 MiB/s wr, 24.18k op/s 2024-03-26T06:41:07.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:07 smithi170 ceph-mon[36912]: pgmap v111: 97 pgs: 97 active+clean; 16 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 383 KiB/s rd, 158 MiB/s wr, 24.18k op/s 2024-03-26T06:41:08.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:07 smithi038 ceph-mon[32265]: pgmap v111: 97 pgs: 97 active+clean; 16 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 383 KiB/s rd, 158 MiB/s wr, 24.18k op/s 2024-03-26T06:41:10.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:09 smithi038 ceph-mon[32265]: pgmap v112: 97 pgs: 97 active+clean; 16 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 383 KiB/s rd, 120 MiB/s wr, 17.11k op/s 2024-03-26T06:41:10.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:09 smithi121 ceph-mon[36349]: pgmap v112: 97 pgs: 97 active+clean; 16 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 383 KiB/s rd, 120 MiB/s wr, 17.11k op/s 2024-03-26T06:41:10.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:09 smithi170 ceph-mon[36912]: pgmap v112: 97 pgs: 97 active+clean; 16 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 383 KiB/s rd, 120 MiB/s wr, 17.11k op/s 2024-03-26T06:41:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.438+0000 7fb677d6f640 1 -- 172.21.15.38:0/1873886620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6701088c0 msgr2=0x7fb670108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.438+0000 7fb677d6f640 1 --2- 172.21.15.38:0/1873886620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6701088c0 0x7fb670108d20 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb66800acb0 tx=0x7fb66802c1c0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.438+0000 7fb677d6f640 1 -- 172.21.15.38:0/1873886620 shutdown_connections 2024-03-26T06:41:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.438+0000 7fb677d6f640 1 --2- 172.21.15.38:0/1873886620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb670113f90 0x7fb6701163a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.438+0000 7fb677d6f640 1 --2- 172.21.15.38:0/1873886620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6701088c0 0x7fb670108d20 secure :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fb66800acb0 tx=0x7fb66802c1c0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.438+0000 7fb677d6f640 1 --2- 172.21.15.38:0/1873886620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb67006d5d0 0x7fb6701082f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.441 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.438+0000 7fb677d6f640 1 -- 172.21.15.38:0/1873886620 >> 172.21.15.38:0/1873886620 conn(0x7fb67006c200 msgr2=0x7fb67006c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:11.449 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.447+0000 7fb677d6f640 1 -- 172.21.15.38:0/1873886620 shutdown_connections 2024-03-26T06:41:11.450 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.448+0000 7fb677d6f640 1 -- 172.21.15.38:0/1873886620 wait complete. 2024-03-26T06:41:11.451 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.448+0000 7fb677d6f640 1 Processor -- start 2024-03-26T06:41:11.453 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.451+0000 7fb677d6f640 1 -- start start 2024-03-26T06:41:11.454 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.451+0000 7fb677d6f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67006d5d0 0x7fb670110c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.452+0000 7fb677d6f640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb670113f90 0x7fb6701111a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.452+0000 7fb677d6f640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6701116e0 0x7fb6701b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.455 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.452+0000 7fb677d6f640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67011bff0 con 0x7fb67006d5d0 2024-03-26T06:41:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.452+0000 7fb677d6f640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67011be70 con 0x7fb6701116e0 2024-03-26T06:41:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.452+0000 7fb677d6f640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67011c170 con 0x7fb670113f90 2024-03-26T06:41:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.452+0000 7fb675b0b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67006d5d0 0x7fb670110c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.452+0000 7fb675b0b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67006d5d0 0x7fb670110c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46594/0 (socket says 172.21.15.38:46594) 2024-03-26T06:41:11.456 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.452+0000 7fb675b0b640 1 -- 172.21.15.38:0/2089124800 learned_addr learned my addr 172.21.15.38:0/2089124800 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.453+0000 7fb675b0b640 1 -- 172.21.15.38:0/2089124800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb670113f90 msgr2=0x7fb6701111a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-26T06:41:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.453+0000 7fb675b0b640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb670113f90 0x7fb6701111a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.453+0000 7fb675b0b640 1 -- 172.21.15.38:0/2089124800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6701116e0 msgr2=0x7fb6701b2d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T06:41:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.453+0000 7fb675b0b640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6701116e0 0x7fb6701b2d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.453+0000 7fb675b0b640 1 -- 172.21.15.38:0/2089124800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6701b32a0 con 0x7fb67006d5d0 2024-03-26T06:41:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.454+0000 7fb675b0b640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67006d5d0 0x7fb670110c60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00b480 tx=0x7fb66c00b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:11.457 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.454+0000 7fb662ffd640 1 -- 172.21.15.38:0/2089124800 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c014070 con 0x7fb67006d5d0 2024-03-26T06:41:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.454+0000 7fb677d6f640 1 -- 172.21.15.38:0/2089124800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6701b3590 con 0x7fb67006d5d0 2024-03-26T06:41:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.454+0000 7fb677d6f640 1 -- 172.21.15.38:0/2089124800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6701b3ab0 con 0x7fb67006d5d0 2024-03-26T06:41:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.455+0000 7fb662ffd640 1 -- 172.21.15.38:0/2089124800 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c010d50 con 0x7fb67006d5d0 2024-03-26T06:41:11.458 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.455+0000 7fb662ffd640 1 -- 172.21.15.38:0/2089124800 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c017360 con 0x7fb67006d5d0 2024-03-26T06:41:11.462 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.459+0000 7fb662ffd640 1 -- 172.21.15.38:0/2089124800 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb66c017070 con 0x7fb67006d5d0 2024-03-26T06:41:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.460+0000 7fb662ffd640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb65c0a2b70 0x7fb65c0a5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.461+0000 7fb662ffd640 1 -- 172.21.15.38:0/2089124800 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fb66c0cf540 con 0x7fb67006d5d0 2024-03-26T06:41:11.464 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.462+0000 7fb67530a640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb65c0a2b70 0x7fb65c0a5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.465 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.463+0000 7fb67aa1f740 1 Processor -- start 2024-03-26T06:41:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.463+0000 7fb67aa1f740 1 -- start start 2024-03-26T06:41:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.463+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81e6c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.463+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e71b0 0x55fba81e7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.463+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.466 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.463+0000 7fb67aa1f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba814b5f0 con 0x55fba81e7f20 2024-03-26T06:41:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.463+0000 7fb67aa1f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba80f4930 con 0x55fba81e71b0 2024-03-26T06:41:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.463+0000 7fb67aa1f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba8147690 con 0x55fba81f0730 2024-03-26T06:41:11.467 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.464+0000 7fb675b0b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81e6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.464+0000 7fb675b0b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81e6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:46608/0 (socket says 172.21.15.38:46608) 2024-03-26T06:41:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.464+0000 7fb675b0b640 1 -- 172.21.15.38:0/3283790549 learned_addr learned my addr 172.21.15.38:0/3283790549 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.464+0000 7fb67530a640 1 --2- 172.21.15.38:0/3283790549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e71b0 0x55fba81e7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.464+0000 7fb67630c640 1 --2- 172.21.15.38:0/3283790549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.464+0000 7fb67530a640 1 -- 172.21.15.38:0/3283790549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 msgr2=0x55fba81f2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.464+0000 7fb67530a640 1 --2- 172.21.15.38:0/3283790549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.464+0000 7fb67530a640 1 -- 172.21.15.38:0/3283790549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 msgr2=0x55fba81e6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.464+0000 7fb67530a640 1 --2- 172.21.15.38:0/3283790549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81e6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.464+0000 7fb67530a640 1 -- 172.21.15.38:0/3283790549 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fba7e60210 con 0x55fba81e71b0 2024-03-26T06:41:11.468 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.465+0000 7fb67630c640 1 --2- 172.21.15.38:0/3283790549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:41:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.465+0000 7fb67530a640 1 --2- 172.21.15.38:0/3283790549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e71b0 0x55fba81e7630 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb670109410 tx=0x7fb6680347a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a611a4c3886c628c server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.466+0000 7fb65bfff640 1 -- 172.21.15.38:0/3283790549 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66804b070 con 0x55fba81e71b0 2024-03-26T06:41:11.469 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.466+0000 7fb65bfff640 1 -- 172.21.15.38:0/3283790549 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb668030070 con 0x55fba81e71b0 2024-03-26T06:41:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.467+0000 7fb67aa1f740 1 -- 172.21.15.38:0/3283790549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e71b0 msgr2=0x55fba81e7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.467+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/3283790549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e71b0 0x55fba81e7630 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb670109410 tx=0x7fb6680347a0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.467+0000 7fb67aa1f740 1 -- 172.21.15.38:0/3283790549 shutdown_connections 2024-03-26T06:41:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.467+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/3283790549 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.467+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/3283790549 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e71b0 0x55fba81e7630 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.467+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/3283790549 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81e6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.470 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.467+0000 7fb67aa1f740 1 -- 172.21.15.38:0/3283790549 >> 172.21.15.38:0/3283790549 conn(0x55fba81ae5b0 msgr2=0x55fba81e1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:11.471 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.469+0000 7fb67530a640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb65c0a2b70 0x7fb65c0a5030 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb66800abd0 tx=0x7fb668045000 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:11.472 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.470+0000 7fb67aa1f740 1 -- 172.21.15.38:0/3283790549 shutdown_connections 2024-03-26T06:41:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.470+0000 7fb67aa1f740 1 -- 172.21.15.38:0/3283790549 wait complete. 2024-03-26T06:41:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.470+0000 7fb67aa1f740 1 -- start start 2024-03-26T06:41:11.473 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.470+0000 7fb67aa1f740 1 Processor -- start 2024-03-26T06:41:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.471+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81f0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.471+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81ed3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.471+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 0x55fba81edde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.471+0000 7fb67aa1f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba81f34b0 con 0x55fba81e7f20 2024-03-26T06:41:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.471+0000 7fb67aa1f740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba8107d90 con 0x55fba81ed980 2024-03-26T06:41:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.471+0000 7fb67aa1f740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fba81f3630 con 0x55fba81f0730 2024-03-26T06:41:11.474 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.471+0000 7fb67aa1f740 10 client.0 _ll_register_callbacks cb 0x55fba7e5fa00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:41:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.472+0000 7fb67530a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81ed3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.472+0000 7fb67530a640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81ed3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:44868/0 (socket says 172.21.15.38:44868) 2024-03-26T06:41:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.472+0000 7fb67530a640 1 -- 172.21.15.38:0/1569566133 learned_addr learned my addr 172.21.15.38:0/1569566133 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.472+0000 7fb67630c640 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 0x55fba81edde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.472+0000 7fb67530a640 1 -- 172.21.15.38:0/1569566133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 msgr2=0x55fba81edde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.472+0000 7fb67530a640 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 0x55fba81edde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.475 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.472+0000 7fb67530a640 1 -- 172.21.15.38:0/1569566133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 msgr2=0x55fba81f0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.472+0000 7fb67530a640 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.472+0000 7fb67530a640 1 -- 172.21.15.38:0/1569566133 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fba7f2c620 con 0x55fba81f0730 2024-03-26T06:41:11.476 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.473+0000 7fb67530a640 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81ed3f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb668035f70 tx=0x7fb66802c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57245f924d6cf7d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:11.491 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.489+0000 7fb6597fa640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-26T06:41:11.492 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.489+0000 7fb6597fa640 1 -- 172.21.15.38:0/1569566133 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668055020 con 0x55fba81f0730 2024-03-26T06:41:11.492 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.489+0000 7fb6597fa640 1 -- 172.21.15.38:0/1569566133 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fb668045890 con 0x55fba81f0730 2024-03-26T06:41:11.492 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.489+0000 7fb67aa1f740 1 -- 172.21.15.38:0/1569566133 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fba8158b30 con 0x55fba81f0730 2024-03-26T06:41:11.492 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.490+0000 7fb6597fa640 1 -- 172.21.15.38:0/1569566133 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb66804b070 con 0x55fba81f0730 2024-03-26T06:41:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.490+0000 7fb67aa1f740 10 client.25138 fetch_fsmap learned FSMap version 27 2024-03-26T06:41:11.493 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.490+0000 7fb67aa1f740 1 -- 172.21.15.38:0/1569566133 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fba81f34e0 con 0x55fba81f0730 2024-03-26T06:41:11.495 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.492+0000 7fb6597fa640 1 -- 172.21.15.38:0/1569566133 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fba8158b30 con 0x55fba81f0730 2024-03-26T06:41:11.496 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.493+0000 7fb6597fa640 1 -- 172.21.15.38:0/1569566133 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fb668043410 con 0x55fba81f0730 2024-03-26T06:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.494+0000 7fb67aa1f740 10 client.25138 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.494+0000 7fb67aa1f740 10 client.25138 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.494+0000 7fb67aa1f740 20 client.25138 populate_metadata read hostname 'smithi038' 2024-03-26T06:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.494+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fba8211f40 0x55fba8214360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.494+0000 7fb67aa1f740 4 client.25138 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.494+0000 7fb67aa1f740 1 -- 172.21.15.38:0/1569566133 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fba8152e80 con 0x55fba8211f40 2024-03-26T06:41:11.497 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.494+0000 7fb675b0b640 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fba8211f40 0x55fba8214360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.495+0000 7fb675b0b640 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fba8211f40 0x55fba8214360 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1eaa84de1f808c5 server_cookie=c27255fe37112d5d in_seq=0 out_seq=0 2024-03-26T06:41:11.498 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.495+0000 7fb6597fa640 10 client.25138 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.504+0000 7fb6597fa640 1 -- 172.21.15.38:0/1569566133 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fba8152e80 con 0x55fba8211f40 2024-03-26T06:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.504+0000 7fb6597fa640 10 client.25138 handle_command_reply: tid=0 2024-03-26T06:41:11.507 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.504+0000 7fb67aa1f740 1 client.25138 shutdown 2024-03-26T06:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.505+0000 7fb67aa1f740 1 -- 172.21.15.38:0/1569566133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 msgr2=0x55fba81ed3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.505+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81ed3f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb668035f70 tx=0x7fb66802c8c0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.505+0000 7fb67aa1f740 1 -- 172.21.15.38:0/1569566133 shutdown_connections 2024-03-26T06:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.505+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fba8211f40 0x55fba8214360 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.505+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 0x55fba81edde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.505+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81ed3f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.505+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/1569566133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81f0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.505+0000 7fb67aa1f740 1 -- 172.21.15.38:0/1569566133 >> 172.21.15.38:0/1569566133 conn(0x55fba81ae5b0 msgr2=0x55fba81e3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:11.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.505+0000 7fb6597fa640 0 client.25138 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.507+0000 7fb67aa1f740 1 -- 172.21.15.38:0/1569566133 shutdown_connections 2024-03-26T06:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.507+0000 7fb67aa1f740 1 -- 172.21.15.38:0/1569566133 wait complete. 2024-03-26T06:41:11.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.507+0000 7fb67aa1f740 20 client.25138 trim_cache size 0 max 16384 2024-03-26T06:41:11.539 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.537+0000 7fb67aa1f740 1 Processor -- start 2024-03-26T06:41:11.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.538+0000 7fb67aa1f740 1 -- start start 2024-03-26T06:41:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.539+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81e8c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.539+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 0x55fba81e19d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.539+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.539+0000 7fb67aa1f740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb654010b60 con 0x55fba81e7f20 2024-03-26T06:41:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.539+0000 7fb67530a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 0x55fba81e19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.539+0000 7fb67530a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 0x55fba81e19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:52324/0 (socket says 172.21.15.38:52324) 2024-03-26T06:41:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.539+0000 7fb67530a640 1 -- 172.21.15.38:0/2262165434 learned_addr learned my addr 172.21.15.38:0/2262165434 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.540+0000 7fb67630c640 1 --2- 172.21.15.38:0/2262165434 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.539+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2262165434 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65400e6a0 con 0x55fba81ed980 2024-03-26T06:41:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.540+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2262165434 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65400c7f0 con 0x55fba81f0730 2024-03-26T06:41:11.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.540+0000 7fb675b0b640 1 --2- 172.21.15.38:0/2262165434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81e8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.540+0000 7fb67530a640 1 -- 172.21.15.38:0/2262165434 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 msgr2=0x55fba81f1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:11.544 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.540+0000 7fb67530a640 1 --2- 172.21.15.38:0/2262165434 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.540+0000 7fb67530a640 1 -- 172.21.15.38:0/2262165434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 msgr2=0x55fba81e8c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.540+0000 7fb67530a640 1 --2- 172.21.15.38:0/2262165434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81e8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.540+0000 7fb67530a640 1 -- 172.21.15.38:0/2262165434 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fba81dc2f0 con 0x55fba81ed980 2024-03-26T06:41:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.541+0000 7fb67530a640 1 --2- 172.21.15.38:0/2262165434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 0x55fba81e19d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb668035ef0 tx=0x7fb6680012e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc5ec08c2fef00a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.541+0000 7fb65b7fe640 1 -- 172.21.15.38:0/2262165434 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668067070 con 0x55fba81ed980 2024-03-26T06:41:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.541+0000 7fb65b7fe640 1 -- 172.21.15.38:0/2262165434 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66804c040 con 0x55fba81ed980 2024-03-26T06:41:11.545 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.541+0000 7fb65b7fe640 1 -- 172.21.15.38:0/2262165434 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb668055040 con 0x55fba81ed980 2024-03-26T06:41:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.542+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2262165434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 msgr2=0x55fba81e19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:11.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.542+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/2262165434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 0x55fba81e19d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb668035ef0 tx=0x7fb6680012e0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.544+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2262165434 shutdown_connections 2024-03-26T06:41:11.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.544+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/2262165434 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.544+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/2262165434 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81ed980 0x55fba81e19d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.544+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/2262165434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81e7f20 0x55fba81e8c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.544+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2262165434 >> 172.21.15.38:0/2262165434 conn(0x55fba81ae5b0 msgr2=0x55fba81e8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.546+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2262165434 shutdown_connections 2024-03-26T06:41:11.549 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.546+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2262165434 wait complete. 2024-03-26T06:41:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.547+0000 7fb67aa1f740 1 -- start start 2024-03-26T06:41:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.547+0000 7fb67aa1f740 1 Processor -- start 2024-03-26T06:41:11.550 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.548+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e7f20 0x55fba81f3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.548+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81ed980 0x55fba81f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.548+0000 7fb675b0b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e7f20 0x55fba81f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.548+0000 7fb675b0b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e7f20 0x55fba81f3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:52340/0 (socket says 172.21.15.38:52340) 2024-03-26T06:41:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.548+0000 7fb67aa1f740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f4380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.549+0000 7fb675b0b640 1 -- 172.21.15.38:0/2801131782 learned_addr learned my addr 172.21.15.38:0/2801131782 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.549+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2801131782 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65400cd60 con 0x55fba81ed980 2024-03-26T06:41:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.549+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2801131782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65400e040 con 0x55fba81e7f20 2024-03-26T06:41:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.549+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2801131782 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65400d380 con 0x55fba81f0730 2024-03-26T06:41:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.549+0000 7fb67630c640 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.552 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.549+0000 7fb67aa1f740 10 client.0 _ll_register_callbacks cb 0x7fb66800ab00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:41:11.553 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.550+0000 7fb67530a640 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81ed980 0x55fba81f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.553 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.550+0000 7fb675b0b640 1 -- 172.21.15.38:0/2801131782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 msgr2=0x55fba81f4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.550+0000 7fb675b0b640 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.550+0000 7fb675b0b640 1 -- 172.21.15.38:0/2801131782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81ed980 msgr2=0x55fba81f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.550+0000 7fb675b0b640 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81ed980 0x55fba81f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.550+0000 7fb675b0b640 1 -- 172.21.15.38:0/2801131782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fba81e6480 con 0x55fba81e7f20 2024-03-26T06:41:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.551+0000 7fb675b0b640 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e7f20 0x55fba81f3900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb66c09bc60 tx=0x7fb66c0cccd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=745ad5e39db6fddb server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:11.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.551+0000 7fb6597fa640 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-03-26T06:41:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.555+0000 7fb6597fa640 1 -- 172.21.15.38:0/2801131782 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c0b8070 con 0x55fba81e7f20 2024-03-26T06:41:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.555+0000 7fb6597fa640 1 -- 172.21.15.38:0/2801131782 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7fb66c098340 con 0x55fba81e7f20 2024-03-26T06:41:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.555+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2801131782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb66804c040 con 0x55fba81e7f20 2024-03-26T06:41:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.556+0000 7fb6597fa640 1 -- 172.21.15.38:0/2801131782 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb66c01e040 con 0x55fba81e7f20 2024-03-26T06:41:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.556+0000 7fb67aa1f740 10 client.25259 fetch_fsmap learned FSMap version 27 2024-03-26T06:41:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.556+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2801131782 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fba8268850 con 0x55fba81e7f20 2024-03-26T06:41:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.557+0000 7fb6597fa640 1 -- 172.21.15.38:0/2801131782 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66804c040 con 0x55fba81e7f20 2024-03-26T06:41:11.562 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.557+0000 7fb6597fa640 1 -- 172.21.15.38:0/2801131782 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 27) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fb66c09b120 con 0x55fba81e7f20 2024-03-26T06:41:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.557+0000 7fb67aa1f740 10 client.25259 fetch_fsmap finished waiting for FSMap version 27 2024-03-26T06:41:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.557+0000 7fb67aa1f740 10 client.25259 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:41:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.557+0000 7fb67aa1f740 20 client.25259 populate_metadata read hostname 'smithi038' 2024-03-26T06:41:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.557+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fba8269020 0x55fba826b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.557+0000 7fb67aa1f740 4 client.25259 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:41:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.557+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2801131782 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fba80d1710 con 0x55fba8269020 2024-03-26T06:41:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.558+0000 7fb67530a640 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fba8269020 0x55fba826b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.559+0000 7fb67530a640 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fba8269020 0x55fba826b440 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9438c6cb6625fce8 server_cookie=93c6b313e51562e8 in_seq=0 out_seq=0 2024-03-26T06:41:11.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:11.559+0000 7fb6597fa640 10 client.25259 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:12.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:11 smithi038 ceph-mon[32265]: pgmap v113: 97 pgs: 97 active+clean; 16 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 462 KiB/s rd, 157 MiB/s wr, 22.47k op/s 2024-03-26T06:41:12.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:11 smithi121 ceph-mon[36349]: pgmap v113: 97 pgs: 97 active+clean; 16 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 462 KiB/s rd, 157 MiB/s wr, 22.47k op/s 2024-03-26T06:41:12.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:11 smithi170 ceph-mon[36912]: pgmap v113: 97 pgs: 97 active+clean; 16 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 462 KiB/s rd, 157 MiB/s wr, 22.47k op/s 2024-03-26T06:41:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb6597fa640 1 -- 172.21.15.38:0/2801131782 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fba80d1710 con 0x55fba8269020 2024-03-26T06:41:12.346 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb6597fa640 10 client.25259 handle_command_reply: tid=0 2024-03-26T06:41:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb67aa1f740 1 client.25259 shutdown 2024-03-26T06:41:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2801131782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e7f20 msgr2=0x55fba81f3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e7f20 0x55fba81f3900 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb66c09bc60 tx=0x7fb66c0cccd0 comp rx=0 tx=0).stop 2024-03-26T06:41:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2801131782 shutdown_connections 2024-03-26T06:41:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fba8269020 0x55fba826b440 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fba81f0730 0x55fba81f4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fba81ed980 0x55fba81f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb67aa1f740 1 --2- 172.21.15.38:0/2801131782 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fba81e7f20 0x55fba81f3900 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:12.347 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2801131782 >> 172.21.15.38:0/2801131782 conn(0x55fba81ae5b0 msgr2=0x55fba81ecc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb6597fa640 0 client.25259 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2801131782 shutdown_connections 2024-03-26T06:41:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.343+0000 7fb67aa1f740 1 -- 172.21.15.38:0/2801131782 wait complete. 2024-03-26T06:41:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.344+0000 7fb67aa1f740 20 client.25259 trim_cache size 0 max 16384 2024-03-26T06:41:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:41:12.348 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:41:12.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:41:12.349 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:41:12.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.344+0000 7fb677d6f640 1 -- 172.21.15.38:0/2089124800 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb65c0a2b70 msgr2=0x7fb65c0a5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:12.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.344+0000 7fb677d6f640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb65c0a2b70 0x7fb65c0a5030 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb66800abd0 tx=0x7fb668045000 comp rx=0 tx=0).stop 2024-03-26T06:41:12.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.344+0000 7fb677d6f640 1 -- 172.21.15.38:0/2089124800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67006d5d0 msgr2=0x7fb670110c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:12.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.344+0000 7fb677d6f640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67006d5d0 0x7fb670110c60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00b480 tx=0x7fb66c00b940 comp rx=0 tx=0).stop 2024-03-26T06:41:12.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.345+0000 7fb677d6f640 1 -- 172.21.15.38:0/2089124800 shutdown_connections 2024-03-26T06:41:12.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.345+0000 7fb677d6f640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb65c0a2b70 0x7fb65c0a5030 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:12.350 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.345+0000 7fb677d6f640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6701116e0 0x7fb6701b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:12.351 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.345+0000 7fb677d6f640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb670113f90 0x7fb6701111a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:12.351 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.345+0000 7fb677d6f640 1 --2- 172.21.15.38:0/2089124800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb67006d5d0 0x7fb670110c60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:12.351 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.345+0000 7fb677d6f640 1 -- 172.21.15.38:0/2089124800 >> 172.21.15.38:0/2089124800 conn(0x7fb67006c200 msgr2=0x7fb67006fbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:12.351 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.346+0000 7fb677d6f640 1 -- 172.21.15.38:0/2089124800 shutdown_connections 2024-03-26T06:41:12.351 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:12.346+0000 7fb677d6f640 1 -- 172.21.15.38:0/2089124800 wait complete. 2024-03-26T06:41:14.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:13 smithi038 ceph-mon[32265]: pgmap v114: 97 pgs: 97 active+clean; 16 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 228 KiB/s rd, 118 MiB/s wr, 16.94k op/s 2024-03-26T06:41:14.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:13 smithi038 ceph-mon[32265]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:41:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:13 smithi121 ceph-mon[36349]: pgmap v114: 97 pgs: 97 active+clean; 16 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 228 KiB/s rd, 118 MiB/s wr, 16.94k op/s 2024-03-26T06:41:14.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:13 smithi121 ceph-mon[36349]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:41:14.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:13 smithi170 ceph-mon[36912]: pgmap v114: 97 pgs: 97 active+clean; 16 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 228 KiB/s rd, 118 MiB/s wr, 16.94k op/s 2024-03-26T06:41:14.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:13 smithi170 ceph-mon[36912]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:41:16.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:15 smithi038 ceph-mon[32265]: pgmap v115: 97 pgs: 97 active+clean; 16 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 229 KiB/s rd, 120 MiB/s wr, 17.42k op/s 2024-03-26T06:41:16.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:15 smithi121 ceph-mon[36349]: pgmap v115: 97 pgs: 97 active+clean; 16 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 229 KiB/s rd, 120 MiB/s wr, 17.42k op/s 2024-03-26T06:41:16.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:15 smithi170 ceph-mon[36912]: pgmap v115: 97 pgs: 97 active+clean; 16 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 229 KiB/s rd, 120 MiB/s wr, 17.42k op/s 2024-03-26T06:41:17.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:16 smithi038 ceph-mon[32265]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:41:17.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:16 smithi121 ceph-mon[36349]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:41:17.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:16 smithi170 ceph-mon[36912]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:41:18.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:17 smithi038 ceph-mon[32265]: pgmap v116: 97 pgs: 97 active+clean; 17 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 265 KiB/s rd, 118 MiB/s wr, 17.69k op/s 2024-03-26T06:41:18.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:17 smithi121 ceph-mon[36349]: pgmap v116: 97 pgs: 97 active+clean; 17 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 265 KiB/s rd, 118 MiB/s wr, 17.69k op/s 2024-03-26T06:41:18.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:17 smithi170 ceph-mon[36912]: pgmap v116: 97 pgs: 97 active+clean; 17 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 265 KiB/s rd, 118 MiB/s wr, 17.69k op/s 2024-03-26T06:41:20.083 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:19 smithi038 ceph-mon[32265]: pgmap v117: 97 pgs: 97 active+clean; 17 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 123 KiB/s rd, 76 MiB/s wr, 11.86k op/s 2024-03-26T06:41:20.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:19 smithi121 ceph-mon[36349]: pgmap v117: 97 pgs: 97 active+clean; 17 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 123 KiB/s rd, 76 MiB/s wr, 11.86k op/s 2024-03-26T06:41:20.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:19 smithi170 ceph-mon[36912]: pgmap v117: 97 pgs: 97 active+clean; 17 GiB data, 75 GiB used, 998 GiB / 1.0 TiB avail; 123 KiB/s rd, 76 MiB/s wr, 11.86k op/s 2024-03-26T06:41:22.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:21 smithi038 ceph-mon[32265]: pgmap v118: 97 pgs: 97 active+clean; 17 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 141 KiB/s rd, 109 MiB/s wr, 17.33k op/s 2024-03-26T06:41:22.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:21 smithi121 ceph-mon[36349]: pgmap v118: 97 pgs: 97 active+clean; 17 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 141 KiB/s rd, 109 MiB/s wr, 17.33k op/s 2024-03-26T06:41:22.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:21 smithi170 ceph-mon[36912]: pgmap v118: 97 pgs: 97 active+clean; 17 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 141 KiB/s rd, 109 MiB/s wr, 17.33k op/s 2024-03-26T06:41:24.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:23 smithi121 ceph-mon[36349]: pgmap v119: 97 pgs: 97 active+clean; 17 GiB data, 76 GiB used, 996 GiB / 1.0 TiB avail; 61 KiB/s rd, 75 MiB/s wr, 12.59k op/s 2024-03-26T06:41:24.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:23 smithi170 ceph-mon[36912]: pgmap v119: 97 pgs: 97 active+clean; 17 GiB data, 76 GiB used, 996 GiB / 1.0 TiB avail; 61 KiB/s rd, 75 MiB/s wr, 12.59k op/s 2024-03-26T06:41:24.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:23 smithi038 ceph-mon[32265]: pgmap v119: 97 pgs: 97 active+clean; 17 GiB data, 76 GiB used, 996 GiB / 1.0 TiB avail; 61 KiB/s rd, 75 MiB/s wr, 12.59k op/s 2024-03-26T06:41:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.498+0000 7f72e4d72640 1 -- 172.21.15.38:0/1429522297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72d80b7a60 msgr2=0x7f72d80b9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.498+0000 7f72e4d72640 1 --2- 172.21.15.38:0/1429522297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72d80b7a60 0x7f72d80b9e50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f72e0066a30 tx=0x7f72e008f0a0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.498+0000 7f72e4d72640 1 -- 172.21.15.38:0/1429522297 shutdown_connections 2024-03-26T06:41:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.498+0000 7f72e4d72640 1 --2- 172.21.15.38:0/1429522297 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72d80b7a60 0x7f72d80b9e50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.508 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.498+0000 7f72e4d72640 1 --2- 172.21.15.38:0/1429522297 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72d80a56f0 0x7f72d80b7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.498+0000 7f72e4d72640 1 --2- 172.21.15.38:0/1429522297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72d80a4dd0 0x7f72d80a51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.498+0000 7f72e4d72640 1 -- 172.21.15.38:0/1429522297 >> 172.21.15.38:0/1429522297 conn(0x7f72d801a630 msgr2=0x7f72d801aa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.498+0000 7f72e4d72640 1 -- 172.21.15.38:0/1429522297 shutdown_connections 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.498+0000 7f72e4d72640 1 -- 172.21.15.38:0/1429522297 wait complete. 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.499+0000 7f72e4d72640 1 Processor -- start 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.499+0000 7f72e4d72640 1 -- start start 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.500+0000 7f72e4d72640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72d80a4dd0 0x7f72d80b7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.500+0000 7f72e4d72640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72d80a56f0 0x7f72d80b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.500+0000 7f72e4d72640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72d80b7a60 0x7f72d80b1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.500+0000 7f72e4d72640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d80ad730 con 0x7f72d80a4dd0 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.500+0000 7f72e4d72640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d80ad5b0 con 0x7f72d80b7a60 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.500+0000 7f72e4d72640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72d80ad8b0 con 0x7f72d80a56f0 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.500+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72d80a4dd0 0x7f72d80b7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:25.509 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.500+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72d80a4dd0 0x7f72d80b7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:59648/0 (socket says 172.21.15.38:59648) 2024-03-26T06:41:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.500+0000 7f72df7fe640 1 -- 172.21.15.38:0/19802131 learned_addr learned my addr 172.21.15.38:0/19802131 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.500+0000 7f72dffff640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72d80b7a60 0x7f72d80b1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.500+0000 7f72deffd640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72d80a56f0 0x7f72d80b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.501+0000 7f72df7fe640 1 -- 172.21.15.38:0/19802131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72d80a56f0 msgr2=0x7f72d80b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.501+0000 7f72df7fe640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72d80a56f0 0x7f72d80b12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.501+0000 7f72df7fe640 1 -- 172.21.15.38:0/19802131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72d80b7a60 msgr2=0x7f72d80b1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.501+0000 7f72df7fe640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72d80b7a60 0x7f72d80b1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.501+0000 7f72df7fe640 1 -- 172.21.15.38:0/19802131 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d80b1fe0 con 0x7f72d80a4dd0 2024-03-26T06:41:25.510 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.502+0000 7f72df7fe640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72d80a4dd0 0x7f72d80b7210 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00e9c0 tx=0x7f72cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:25.511 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.507+0000 7f72dcff9640 1 -- 172.21.15.38:0/19802131 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc00cd80 con 0x7f72d80a4dd0 2024-03-26T06:41:25.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.507+0000 7f72e4d72640 1 -- 172.21.15.38:0/19802131 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72d800bc30 con 0x7f72d80a4dd0 2024-03-26T06:41:25.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.507+0000 7f72e4d72640 1 -- 172.21.15.38:0/19802131 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72d800c180 con 0x7f72d80a4dd0 2024-03-26T06:41:25.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.510+0000 7f72dcff9640 1 -- 172.21.15.38:0/19802131 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72cc0035f0 con 0x7f72d80a4dd0 2024-03-26T06:41:25.513 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.510+0000 7f72dcff9640 1 -- 172.21.15.38:0/19802131 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc010690 con 0x7f72d80a4dd0 2024-03-26T06:41:25.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.511+0000 7f72e8bb4740 1 Processor -- start 2024-03-26T06:41:25.514 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.511+0000 7f72e8bb4740 1 -- start start 2024-03-26T06:41:25.515 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.511+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d1f20 0x55fc211d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.515 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.511+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.511+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211da730 0x55fc211dcb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.511+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc211355f0 con 0x55fc211d1f20 2024-03-26T06:41:25.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.511+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc210de930 con 0x55fc211da730 2024-03-26T06:41:25.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.511+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc21131690 con 0x55fc211d11b0 2024-03-26T06:41:25.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.512+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:25.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.512+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:36118/0 (socket says 172.21.15.38:36118) 2024-03-26T06:41:25.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.512+0000 7f72df7fe640 1 -- 172.21.15.38:0/1183907277 learned_addr learned my addr 172.21.15.38:0/1183907277 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:25.516 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.512+0000 7f72df7fe640 1 -- 172.21.15.38:0/1183907277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211da730 msgr2=0x55fc211dcb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:25.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.512+0000 7f72df7fe640 1 --2- 172.21.15.38:0/1183907277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211da730 0x55fc211dcb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.512+0000 7f72df7fe640 1 -- 172.21.15.38:0/1183907277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d1f20 msgr2=0x55fc211d0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:25.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.512+0000 7f72df7fe640 1 --2- 172.21.15.38:0/1183907277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d1f20 0x55fc211d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.517 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.512+0000 7f72df7fe640 1 -- 172.21.15.38:0/1183907277 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc20e4a210 con 0x55fc211d11b0 2024-03-26T06:41:25.518 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.512+0000 7f72df7fe640 1 --2- 172.21.15.38:0/1183907277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d1630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009830 tx=0x7f72cc003d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d9aa77f326c0ff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.513+0000 7f72c9ffb640 1 -- 172.21.15.38:0/1183907277 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc031070 con 0x55fc211d11b0 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.513+0000 7f72c9ffb640 1 -- 172.21.15.38:0/1183907277 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72cc005510 con 0x55fc211d11b0 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.514+0000 7f72e8bb4740 1 -- 172.21.15.38:0/1183907277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 msgr2=0x55fc211d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.514+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/1183907277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d1630 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f72cc009830 tx=0x7f72cc003d20 comp rx=0 tx=0).stop 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.515+0000 7f72dcff9640 1 -- 172.21.15.38:0/19802131 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f72cc021430 con 0x7f72d80a4dd0 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.515+0000 7f72e8bb4740 1 -- 172.21.15.38:0/1183907277 shutdown_connections 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.515+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/1183907277 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211da730 0x55fc211dcb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.515+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/1183907277 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d1630 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.515+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/1183907277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d1f20 0x55fc211d0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.515+0000 7f72e8bb4740 1 -- 172.21.15.38:0/1183907277 >> 172.21.15.38:0/1183907277 conn(0x55fc211985b0 msgr2=0x55fc211cbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.515+0000 7f72e8bb4740 1 -- 172.21.15.38:0/1183907277 shutdown_connections 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.515+0000 7f72e8bb4740 1 -- 172.21.15.38:0/1183907277 wait complete. 2024-03-26T06:41:25.519 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.515+0000 7f72e8bb4740 1 -- start start 2024-03-26T06:41:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.516+0000 7f72e8bb4740 1 Processor -- start 2024-03-26T06:41:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.516+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d11b0 0x55fc211d76c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.516+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d1f20 0x55fc211d7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.516+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 0x55fc211d8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.516+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc211dd400 con 0x55fc211d11b0 2024-03-26T06:41:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.516+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc210f1d90 con 0x55fc211d1f20 2024-03-26T06:41:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.516+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fc211dd580 con 0x55fc211da730 2024-03-26T06:41:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.516+0000 7f72e8bb4740 10 client.0 _ll_register_callbacks cb 0x55fc20e49a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:41:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.516+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d1f20 0x55fc211d7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.517+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d1f20 0x55fc211d7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:47156/0 (socket says 172.21.15.38:47156) 2024-03-26T06:41:25.521 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.517+0000 7f72df7fe640 1 -- 172.21.15.38:0/2475081468 learned_addr learned my addr 172.21.15.38:0/2475081468 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:25.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.516+0000 7f72deffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 0x55fc211d8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:25.522 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.517+0000 7f72df7fe640 1 -- 172.21.15.38:0/2475081468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 msgr2=0x55fc211d8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.517+0000 7f72df7fe640 1 --2- 172.21.15.38:0/2475081468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 0x55fc211d8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.517+0000 7f72df7fe640 1 -- 172.21.15.38:0/2475081468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d11b0 msgr2=0x55fc211d76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.517+0000 7f72df7fe640 1 --2- 172.21.15.38:0/2475081468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d11b0 0x55fc211d76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.517+0000 7f72df7fe640 1 -- 172.21.15.38:0/2475081468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc20f16620 con 0x55fc211d1f20 2024-03-26T06:41:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.517+0000 7f72df7fe640 1 --2- 172.21.15.38:0/2475081468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d1f20 0x55fc211d7c00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f72cc01c000 tx=0x7f72cc020000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66b8a683ae971e1f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:25.523 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.518+0000 7f72beffd640 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-03-26T06:41:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.518+0000 7f72beffd640 1 -- 172.21.15.38:0/2475081468 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc031070 con 0x55fc211d1f20 2024-03-26T06:41:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.519+0000 7f72dcff9640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72d00a29b0 0x7f72d00a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.519+0000 7f72e8bb4740 1 -- 172.21.15.38:0/2475081468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72cc005510 con 0x55fc211d1f20 2024-03-26T06:41:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.519+0000 7f72beffd640 1 -- 172.21.15.38:0/2475081468 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f72cc0dc830 con 0x55fc211d1f20 2024-03-26T06:41:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.519+0000 7f72beffd640 1 -- 172.21.15.38:0/2475081468 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72cc0ed050 con 0x55fc211d1f20 2024-03-26T06:41:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.519+0000 7f72e8bb4740 10 client.25268 fetch_fsmap learned FSMap version 29 2024-03-26T06:41:25.524 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.519+0000 7f72e8bb4740 1 -- 172.21.15.38:0/2475081468 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55fc211e9720 con 0x55fc211d1f20 2024-03-26T06:41:25.525 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.520+0000 7f72beffd640 1 -- 172.21.15.38:0/2475081468 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72cc005510 con 0x55fc211d1f20 2024-03-26T06:41:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.521+0000 7f72dffff640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72d00a29b0 0x7f72d00a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.521+0000 7f72beffd640 1 -- 172.21.15.38:0/2475081468 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 29) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f72cc09bc40 con 0x55fc211d1f20 2024-03-26T06:41:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.522+0000 7f72e8bb4740 10 client.25268 fetch_fsmap finished waiting for FSMap version 29 2024-03-26T06:41:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.522+0000 7f72e8bb4740 10 client.25268 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:41:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.522+0000 7f72e8bb4740 20 client.25268 populate_metadata read hostname 'smithi038' 2024-03-26T06:41:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.522+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/2475081468 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fc211eaf10 0x55fc211ed330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.522+0000 7f72e8bb4740 4 client.25268 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:41:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.522+0000 7f72e8bb4740 1 -- 172.21.15.38:0/2475081468 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fc21142b30 con 0x55fc211eaf10 2024-03-26T06:41:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.522+0000 7f72deffd640 1 --2- 172.21.15.38:0/2475081468 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fc211eaf10 0x55fc211ed330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.522+0000 7f72dcff9640 1 -- 172.21.15.38:0/19802131 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f72cc0d2f50 con 0x7f72d80a4dd0 2024-03-26T06:41:25.526 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.523+0000 7f72dffff640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72d00a29b0 0x7f72d00a4e70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f72e0064430 tx=0x7f72e009a040 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:25.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.524+0000 7f72deffd640 1 --2- 172.21.15.38:0/2475081468 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fc211eaf10 0x55fc211ed330 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2200e0b8473fc35d server_cookie=89372cdae5e96099 in_seq=0 out_seq=0 2024-03-26T06:41:25.527 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.524+0000 7f72beffd640 10 client.25268 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:25.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.526+0000 7f72beffd640 1 -- 172.21.15.38:0/2475081468 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fc21142b30 con 0x55fc211eaf10 2024-03-26T06:41:25.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.526+0000 7f72beffd640 10 client.25268 handle_command_reply: tid=0 2024-03-26T06:41:25.528 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.526+0000 7f72e8bb4740 1 client.25268 shutdown 2024-03-26T06:41:25.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.526+0000 7f72e8bb4740 1 -- 172.21.15.38:0/2475081468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d1f20 msgr2=0x55fc211d7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:25.529 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.526+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/2475081468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d1f20 0x55fc211d7c00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f72cc01c000 tx=0x7f72cc020000 comp rx=0 tx=0).stop 2024-03-26T06:41:25.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.529+0000 7f72e8bb4740 1 -- 172.21.15.38:0/2475081468 shutdown_connections 2024-03-26T06:41:25.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.529+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/2475081468 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fc211eaf10 0x55fc211ed330 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.529+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/2475081468 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 0x55fc211d8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.529+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/2475081468 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d1f20 0x55fc211d7c00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.529+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/2475081468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d11b0 0x55fc211d76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.532 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.529+0000 7f72e8bb4740 1 -- 172.21.15.38:0/2475081468 >> 172.21.15.38:0/2475081468 conn(0x55fc211985b0 msgr2=0x55fc211cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:25.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.529+0000 7f72beffd640 0 client.25268 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:25.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.530+0000 7f72e8bb4740 1 -- 172.21.15.38:0/2475081468 shutdown_connections 2024-03-26T06:41:25.533 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.530+0000 7f72e8bb4740 1 -- 172.21.15.38:0/2475081468 wait complete. 2024-03-26T06:41:25.535 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.533+0000 7f72e8bb4740 20 client.25268 trim_cache size 0 max 16384 2024-03-26T06:41:25.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.564+0000 7f72e8bb4740 1 Processor -- start 2024-03-26T06:41:25.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.564+0000 7f72e8bb4740 1 -- start start 2024-03-26T06:41:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d1f20 0x55fc211f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211da730 0x55fc211d5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c4010a30 con 0x55fc211da730 2024-03-26T06:41:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c400e5a0 con 0x55fc211d1f20 2024-03-26T06:41:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c400c6f0 con 0x55fc211d11b0 2024-03-26T06:41:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:25.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:36142/0 (socket says 172.21.15.38:36142) 2024-03-26T06:41:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72df7fe640 1 -- 172.21.15.38:0/1665238881 learned_addr learned my addr 172.21.15.38:0/1665238881 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72df7fe640 1 -- 172.21.15.38:0/1665238881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d1f20 msgr2=0x55fc211f1c50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-26T06:41:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72df7fe640 1 --2- 172.21.15.38:0/1665238881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d1f20 0x55fc211f1c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72df7fe640 1 -- 172.21.15.38:0/1665238881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211da730 msgr2=0x55fc211d5f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72df7fe640 1 --2- 172.21.15.38:0/1665238881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211da730 0x55fc211d5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.565+0000 7f72df7fe640 1 -- 172.21.15.38:0/1665238881 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fc211c62f0 con 0x55fc211d11b0 2024-03-26T06:41:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.566+0000 7f72df7fe640 1 --2- 172.21.15.38:0/1665238881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d9df0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f72cc005820 tx=0x7f72cc004a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63b21954f4738014 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:25.570 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.567+0000 7f72c97fa640 1 -- 172.21.15.38:0/1665238881 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc09f110 con 0x55fc211d11b0 2024-03-26T06:41:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.567+0000 7f72c97fa640 1 -- 172.21.15.38:0/1665238881 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72cc09b890 con 0x55fc211d11b0 2024-03-26T06:41:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.567+0000 7f72c97fa640 1 -- 172.21.15.38:0/1665238881 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc0a7ce0 con 0x55fc211d11b0 2024-03-26T06:41:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.568+0000 7f72e8bb4740 1 -- 172.21.15.38:0/1665238881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 msgr2=0x55fc211d9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:25.571 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.568+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/1665238881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d9df0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f72cc005820 tx=0x7f72cc004a20 comp rx=0 tx=0).stop 2024-03-26T06:41:25.574 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.571+0000 7f72e8bb4740 1 -- 172.21.15.38:0/1665238881 shutdown_connections 2024-03-26T06:41:25.574 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.571+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/1665238881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211da730 0x55fc211d5f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.571+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/1665238881 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d1f20 0x55fc211f1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.571+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/1665238881 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211d11b0 0x55fc211d9df0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.571+0000 7f72e8bb4740 1 -- 172.21.15.38:0/1665238881 >> 172.21.15.38:0/1665238881 conn(0x55fc211985b0 msgr2=0x55fc211c6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.571+0000 7f72e8bb4740 1 -- 172.21.15.38:0/1665238881 shutdown_connections 2024-03-26T06:41:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.572+0000 7f72e8bb4740 1 -- 172.21.15.38:0/1665238881 wait complete. 2024-03-26T06:41:25.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.572+0000 7f72e8bb4740 1 -- start start 2024-03-26T06:41:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.573+0000 7f72e8bb4740 1 Processor -- start 2024-03-26T06:41:25.576 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.573+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d11b0 0x55fc211df8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.573+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d1f20 0x55fc211dfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.573+0000 7f72e8bb4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 0x55fc211e0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.574+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c400cc60 con 0x55fc211d1f20 2024-03-26T06:41:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.574+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c400df40 con 0x55fc211d11b0 2024-03-26T06:41:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.574+0000 7f72e8bb4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72c400d280 con 0x55fc211da730 2024-03-26T06:41:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.574+0000 7f72dffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 0x55fc211e0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.574+0000 7f72dffff640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 0x55fc211e0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:36152/0 (socket says 172.21.15.38:36152) 2024-03-26T06:41:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.574+0000 7f72dffff640 1 -- 172.21.15.38:0/4124165772 learned_addr learned my addr 172.21.15.38:0/4124165772 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:25.577 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.574+0000 7f72e8bb4740 10 client.0 _ll_register_callbacks cb 0x7f72cc025040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:41:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.574+0000 7f72dffff640 1 -- 172.21.15.38:0/4124165772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d11b0 msgr2=0x55fc211df8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-26T06:41:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.574+0000 7f72dffff640 1 --2- 172.21.15.38:0/4124165772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d11b0 0x55fc211df8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.574+0000 7f72dffff640 1 -- 172.21.15.38:0/4124165772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d1f20 msgr2=0x55fc211dfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.575+0000 7f72dffff640 1 --2- 172.21.15.38:0/4124165772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d1f20 0x55fc211dfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:25.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.575+0000 7f72dffff640 1 -- 172.21.15.38:0/4124165772 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fc211d0480 con 0x55fc211da730 2024-03-26T06:41:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.579+0000 7f72dffff640 1 --2- 172.21.15.38:0/4124165772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 0x55fc211e0340 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f72e0066fd0 tx=0x7f72e0067410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26cec4858655ca32 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.579+0000 7f72beffd640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-26T06:41:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.579+0000 7f72beffd640 1 -- 172.21.15.38:0/4124165772 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e00b2060 con 0x55fc211da730 2024-03-26T06:41:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.579+0000 7f72e8bb4740 1 -- 172.21.15.38:0/4124165772 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72cc09b890 con 0x55fc211da730 2024-03-26T06:41:25.582 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.579+0000 7f72beffd640 1 -- 172.21.15.38:0/4124165772 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f72e0098670 con 0x55fc211da730 2024-03-26T06:41:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.580+0000 7f72beffd640 1 -- 172.21.15.38:0/4124165772 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72e0098870 con 0x55fc211da730 2024-03-26T06:41:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.580+0000 7f72e8bb4740 10 client.25162 fetch_fsmap learned FSMap version 29 2024-03-26T06:41:25.583 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.580+0000 7f72e8bb4740 1 -- 172.21.15.38:0/4124165772 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55fc21241790 con 0x55fc211da730 2024-03-26T06:41:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.584+0000 7f72beffd640 1 -- 172.21.15.38:0/4124165772 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72e0093040 con 0x55fc211da730 2024-03-26T06:41:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.584+0000 7f72beffd640 1 -- 172.21.15.38:0/4124165772 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 29) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f72e00a2c20 con 0x55fc211da730 2024-03-26T06:41:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.584+0000 7f72e8bb4740 10 client.25162 fetch_fsmap finished waiting for FSMap version 29 2024-03-26T06:41:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.584+0000 7f72e8bb4740 10 client.25162 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:41:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.584+0000 7f72e8bb4740 20 client.25162 populate_metadata read hostname 'smithi038' 2024-03-26T06:41:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.584+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/4124165772 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fc21241f60 0x55fc21244380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:25.587 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.584+0000 7f72e8bb4740 4 client.25162 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:41:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.584+0000 7f72e8bb4740 1 -- 172.21.15.38:0/4124165772 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7f72cc09b890 con 0x55fc21241f60 2024-03-26T06:41:25.588 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.585+0000 7f72df7fe640 1 --2- 172.21.15.38:0/4124165772 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fc21241f60 0x55fc21244380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.587+0000 7f72df7fe640 1 --2- 172.21.15.38:0/4124165772 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fc21241f60 0x55fc21244380 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3df6767308154a6 server_cookie=cca64e68c235861d in_seq=0 out_seq=0 2024-03-26T06:41:25.590 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:25.587+0000 7f72beffd640 10 client.25162 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:26.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.031+0000 7f72beffd640 1 -- 172.21.15.38:0/4124165772 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f72cc09b890 con 0x55fc21241f60 2024-03-26T06:41:26.035 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.031+0000 7f72beffd640 10 client.25162 handle_command_reply: tid=0 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 1 client.25162 shutdown 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 1 -- 172.21.15.38:0/4124165772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 msgr2=0x55fc211e0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/4124165772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 0x55fc211e0340 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f72e0066fd0 tx=0x7f72e0067410 comp rx=0 tx=0).stop 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 1 -- 172.21.15.38:0/4124165772 shutdown_connections 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/4124165772 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x55fc21241f60 0x55fc21244380 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/4124165772 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x55fc211da730 0x55fc211e0340 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/4124165772 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55fc211d1f20 0x55fc211dfe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 1 --2- 172.21.15.38:0/4124165772 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x55fc211d11b0 0x55fc211df8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 1 -- 172.21.15.38:0/4124165772 >> 172.21.15.38:0/4124165772 conn(0x55fc211985b0 msgr2=0x55fc211d06d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72beffd640 0 client.25162 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 1 -- 172.21.15.38:0/4124165772 shutdown_connections 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 1 -- 172.21.15.38:0/4124165772 wait complete. 2024-03-26T06:41:26.036 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.032+0000 7f72e8bb4740 20 client.25162 trim_cache size 0 max 16384 2024-03-26T06:41:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:41:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:41:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:41:26.037 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:41:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.033+0000 7f72e4d72640 1 -- 172.21.15.38:0/19802131 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72d00a29b0 msgr2=0x7f72d00a4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.033+0000 7f72e4d72640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72d00a29b0 0x7f72d00a4e70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f72e0064430 tx=0x7f72e009a040 comp rx=0 tx=0).stop 2024-03-26T06:41:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.033+0000 7f72e4d72640 1 -- 172.21.15.38:0/19802131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72d80a4dd0 msgr2=0x7f72d80b7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.033+0000 7f72e4d72640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72d80a4dd0 0x7f72d80b7210 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f72cc00e9c0 tx=0x7f72cc00ee80 comp rx=0 tx=0).stop 2024-03-26T06:41:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.033+0000 7f72e4d72640 1 -- 172.21.15.38:0/19802131 shutdown_connections 2024-03-26T06:41:26.038 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.034+0000 7f72e4d72640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72d00a29b0 0x7f72d00a4e70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.034+0000 7f72e4d72640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72d80b7a60 0x7f72d80b1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.034+0000 7f72e4d72640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72d80a56f0 0x7f72d80b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.034+0000 7f72e4d72640 1 --2- 172.21.15.38:0/19802131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72d80a4dd0 0x7f72d80b7210 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.034+0000 7f72e4d72640 1 -- 172.21.15.38:0/19802131 >> 172.21.15.38:0/19802131 conn(0x7f72d801a630 msgr2=0x7f72d80b8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.034+0000 7f72e4d72640 1 -- 172.21.15.38:0/19802131 shutdown_connections 2024-03-26T06:41:26.039 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:26.034+0000 7f72e4d72640 1 -- 172.21.15.38:0/19802131 wait complete. 2024-03-26T06:41:26.135 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:25 smithi038 ceph-mon[32265]: pgmap v120: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 62 KiB/s rd, 76 MiB/s wr, 13.20k op/s 2024-03-26T06:41:26.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:25 smithi121 ceph-mon[36349]: pgmap v120: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 62 KiB/s rd, 76 MiB/s wr, 13.20k op/s 2024-03-26T06:41:26.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:25 smithi170 ceph-mon[36912]: pgmap v120: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 62 KiB/s rd, 76 MiB/s wr, 13.20k op/s 2024-03-26T06:41:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:27 smithi121 ceph-mon[36349]: pgmap v121: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 83 KiB/s rd, 98 MiB/s wr, 17.43k op/s 2024-03-26T06:41:28.167 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:27 smithi170 ceph-mon[36912]: pgmap v121: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 83 KiB/s rd, 98 MiB/s wr, 17.43k op/s 2024-03-26T06:41:28.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:27 smithi038 ceph-mon[32265]: pgmap v121: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 83 KiB/s rd, 98 MiB/s wr, 17.43k op/s 2024-03-26T06:41:30.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:29 smithi121 ceph-mon[36349]: pgmap v122: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 40 KiB/s rd, 84 MiB/s wr, 14.92k op/s 2024-03-26T06:41:30.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:29 smithi170 ceph-mon[36912]: pgmap v122: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 40 KiB/s rd, 84 MiB/s wr, 14.92k op/s 2024-03-26T06:41:30.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:29 smithi038 ceph-mon[32265]: pgmap v122: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 40 KiB/s rd, 84 MiB/s wr, 14.92k op/s 2024-03-26T06:41:32.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:31 smithi121 ceph-mon[36349]: pgmap v123: 97 pgs: 97 active+clean; 18 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 75 KiB/s rd, 102 MiB/s wr, 18.60k op/s 2024-03-26T06:41:32.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:31 smithi170 ceph-mon[36912]: pgmap v123: 97 pgs: 97 active+clean; 18 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 75 KiB/s rd, 102 MiB/s wr, 18.60k op/s 2024-03-26T06:41:32.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:31 smithi038 ceph-mon[32265]: pgmap v123: 97 pgs: 97 active+clean; 18 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 75 KiB/s rd, 102 MiB/s wr, 18.60k op/s 2024-03-26T06:41:34.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:33 smithi121 ceph-mon[36349]: pgmap v124: 97 pgs: 97 active+clean; 18 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 58 KiB/s rd, 74 MiB/s wr, 13.98k op/s 2024-03-26T06:41:34.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:33 smithi170 ceph-mon[36912]: pgmap v124: 97 pgs: 97 active+clean; 18 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 58 KiB/s rd, 74 MiB/s wr, 13.98k op/s 2024-03-26T06:41:34.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:33 smithi038 ceph-mon[32265]: pgmap v124: 97 pgs: 97 active+clean; 18 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 58 KiB/s rd, 74 MiB/s wr, 13.98k op/s 2024-03-26T06:41:36.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:35 smithi121 ceph-mon[36349]: pgmap v125: 97 pgs: 97 active+clean; 18 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 58 KiB/s rd, 72 MiB/s wr, 13.68k op/s 2024-03-26T06:41:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:35 smithi170 ceph-mon[36912]: pgmap v125: 97 pgs: 97 active+clean; 18 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 58 KiB/s rd, 72 MiB/s wr, 13.68k op/s 2024-03-26T06:41:36.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:35 smithi038 ceph-mon[32265]: pgmap v125: 97 pgs: 97 active+clean; 18 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 58 KiB/s rd, 72 MiB/s wr, 13.68k op/s 2024-03-26T06:41:38.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:37 smithi121 ceph-mon[36349]: pgmap v126: 97 pgs: 97 active+clean; 18 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 85 KiB/s rd, 82 MiB/s wr, 15.26k op/s 2024-03-26T06:41:38.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:37 smithi170 ceph-mon[36912]: pgmap v126: 97 pgs: 97 active+clean; 18 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 85 KiB/s rd, 82 MiB/s wr, 15.26k op/s 2024-03-26T06:41:38.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:37 smithi038 ceph-mon[32265]: pgmap v126: 97 pgs: 97 active+clean; 18 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 85 KiB/s rd, 82 MiB/s wr, 15.26k op/s 2024-03-26T06:41:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.212+0000 7f2ca8e12640 1 -- 172.21.15.38:0/2455443206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ca41062b0 msgr2=0x7f2ca4068970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.212+0000 7f2ca8e12640 1 --2- 172.21.15.38:0/2455443206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ca41062b0 0x7f2ca4068970 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2c9401a910 tx=0x7f2c9403cd20 comp rx=0 tx=0).stop 2024-03-26T06:41:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.213+0000 7f2ca8e12640 1 -- 172.21.15.38:0/2455443206 shutdown_connections 2024-03-26T06:41:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.213+0000 7f2ca8e12640 1 --2- 172.21.15.38:0/2455443206 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ca4103d00 0x7f2ca4104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.213+0000 7f2ca8e12640 1 --2- 172.21.15.38:0/2455443206 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ca4068eb0 0x7f2ca40692f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.213+0000 7f2ca8e12640 1 --2- 172.21.15.38:0/2455443206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ca41062b0 0x7f2ca4068970 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.216 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.213+0000 7f2ca8e12640 1 -- 172.21.15.38:0/2455443206 >> 172.21.15.38:0/2455443206 conn(0x7f2ca4077040 msgr2=0x7f2ca4077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:39.232 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.229+0000 7f2ca8e12640 1 -- 172.21.15.38:0/2455443206 shutdown_connections 2024-03-26T06:41:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.229+0000 7f2ca8e12640 1 -- 172.21.15.38:0/2455443206 wait complete. 2024-03-26T06:41:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.231+0000 7f2ca8e12640 1 Processor -- start 2024-03-26T06:41:39.233 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.231+0000 7f2ca8e12640 1 -- start start 2024-03-26T06:41:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.231+0000 7f2ca8e12640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ca4068eb0 0x7f2ca4071cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.231+0000 7f2ca8e12640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ca4103d00 0x7f2ca40721f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.231+0000 7f2ca8e12640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ca41062b0 0x7f2ca410cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.231+0000 7f2ca8e12640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca41094b0 con 0x7f2ca4068eb0 2024-03-26T06:41:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.231+0000 7f2ca8e12640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca4109330 con 0x7f2ca41062b0 2024-03-26T06:41:39.234 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.231+0000 7f2ca8e12640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca4109630 con 0x7f2ca4103d00 2024-03-26T06:41:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.231+0000 7f2ca259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ca4068eb0 0x7f2ca4071cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.231+0000 7f2ca259c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ca4068eb0 0x7f2ca4071cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39108/0 (socket says 172.21.15.38:39108) 2024-03-26T06:41:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.231+0000 7f2ca259c640 1 -- 172.21.15.38:0/3198841832 learned_addr learned my addr 172.21.15.38:0/3198841832 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.232+0000 7f2ca259c640 1 -- 172.21.15.38:0/3198841832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ca4103d00 msgr2=0x7f2ca40721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.232+0000 7f2ca259c640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ca4103d00 0x7f2ca40721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.232+0000 7f2ca259c640 1 -- 172.21.15.38:0/3198841832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ca41062b0 msgr2=0x7f2ca410cd40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-26T06:41:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.232+0000 7f2ca259c640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ca41062b0 0x7f2ca410cd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.235 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.232+0000 7f2ca259c640 1 -- 172.21.15.38:0/3198841832 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca410d280 con 0x7f2ca4068eb0 2024-03-26T06:41:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.233+0000 7f2ca259c640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ca4068eb0 0x7f2ca4071cb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2c9403d330 tx=0x7f2c940045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:39.236 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.233+0000 7f2c937fe640 1 -- 172.21.15.38:0/3198841832 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94052020 con 0x7f2ca4068eb0 2024-03-26T06:41:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.234+0000 7f2ca8e12640 1 -- 172.21.15.38:0/3198841832 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca410d510 con 0x7f2ca4068eb0 2024-03-26T06:41:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.234+0000 7f2ca8e12640 1 -- 172.21.15.38:0/3198841832 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca410dae0 con 0x7f2ca4068eb0 2024-03-26T06:41:39.239 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.236+0000 7f2c937fe640 1 -- 172.21.15.38:0/3198841832 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c9403ddb0 con 0x7f2ca4068eb0 2024-03-26T06:41:39.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.237+0000 7f2c937fe640 1 -- 172.21.15.38:0/3198841832 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94003db0 con 0x7f2ca4068eb0 2024-03-26T06:41:39.240 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.238+0000 7f2cab9f4740 1 Processor -- start 2024-03-26T06:41:39.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.238+0000 7f2cab9f4740 1 -- start start 2024-03-26T06:41:39.241 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.238+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2454d40 0x5649c2453a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.238+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2453fd0 0x5649c2454450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.238+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 0x5649c245f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.238+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649c23b85f0 con 0x5649c2454d40 2024-03-26T06:41:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.238+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649c2361930 con 0x5649c2453fd0 2024-03-26T06:41:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.238+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649c23b4690 con 0x5649c245d550 2024-03-26T06:41:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.239+0000 7f2ca259c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2453fd0 0x5649c2454450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.239+0000 7f2ca259c640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2453fd0 0x5649c2454450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:55736/0 (socket says 172.21.15.38:55736) 2024-03-26T06:41:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.239+0000 7f2ca259c640 1 -- 172.21.15.38:0/4113628170 learned_addr learned my addr 172.21.15.38:0/4113628170 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.239+0000 7f2ca259c640 1 -- 172.21.15.38:0/4113628170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 msgr2=0x5649c245f940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-26T06:41:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.239+0000 7f2ca259c640 1 --2- 172.21.15.38:0/4113628170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 0x5649c245f940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.242 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.239+0000 7f2ca259c640 1 -- 172.21.15.38:0/4113628170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2454d40 msgr2=0x5649c2453a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.239+0000 7f2ca259c640 1 --2- 172.21.15.38:0/4113628170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2454d40 0x5649c2453a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.243 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.239+0000 7f2ca259c640 1 -- 172.21.15.38:0/4113628170 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649c20cd210 con 0x5649c2453fd0 2024-03-26T06:41:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.241+0000 7f2c937fe640 1 -- 172.21.15.38:0/3198841832 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2c94055020 con 0x7f2ca4068eb0 2024-03-26T06:41:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.241+0000 7f2ca259c640 1 --2- 172.21.15.38:0/4113628170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2453fd0 0x5649c2454450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2c94006010 tx=0x7f2c9401a620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e9eff28b6892482 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:39.244 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.242+0000 7f2c937fe640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f2c8c0a2c40 0x7f2c8c0a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.245 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.242+0000 7f2c937fe640 1 -- 172.21.15.38:0/3198841832 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f2c94068070 con 0x7f2ca4068eb0 2024-03-26T06:41:39.246 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.243+0000 7f2ca2d9d640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f2c8c0a2c40 0x7f2c8c0a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.245+0000 7f2c90ff9640 1 -- 172.21.15.38:0/4113628170 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94105650 con 0x5649c2453fd0 2024-03-26T06:41:39.248 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.245+0000 7f2c90ff9640 1 -- 172.21.15.38:0/4113628170 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c9410a020 con 0x5649c2453fd0 2024-03-26T06:41:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.246+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4113628170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2453fd0 msgr2=0x5649c2454450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:39.249 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.246+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/4113628170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2453fd0 0x5649c2454450 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2c94006010 tx=0x7f2c9401a620 comp rx=0 tx=0).stop 2024-03-26T06:41:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.248+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4113628170 shutdown_connections 2024-03-26T06:41:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.248+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/4113628170 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 0x5649c245f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.248+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/4113628170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2453fd0 0x5649c2454450 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.251 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.248+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/4113628170 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2454d40 0x5649c2453a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.248+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4113628170 >> 172.21.15.38:0/4113628170 conn(0x5649c241b5b0 msgr2=0x5649c244ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:39.252 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.250+0000 7f2ca2d9d640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f2c8c0a2c40 0x7f2c8c0a5100 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c001f30 tx=0x7f2c9c001610 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.250+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4113628170 shutdown_connections 2024-03-26T06:41:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.250+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4113628170 wait complete. 2024-03-26T06:41:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.251+0000 7f2cab9f4740 1 -- start start 2024-03-26T06:41:39.253 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.251+0000 7f2cab9f4740 1 Processor -- start 2024-03-26T06:41:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.251+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 0x5649c245a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.252+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c2454d40 0x5649c245aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.254 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.252+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c245d550 0x5649c245b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.252+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649c2460040 con 0x5649c2453fd0 2024-03-26T06:41:39.255 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.252+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649c2374d90 con 0x5649c245d550 2024-03-26T06:41:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.252+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x5649c24601c0 con 0x5649c2454d40 2024-03-26T06:41:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.252+0000 7f2ca2d9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c245d550 0x5649c245b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.252+0000 7f2ca2d9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c245d550 0x5649c245b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:55744/0 (socket says 172.21.15.38:55744) 2024-03-26T06:41:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.252+0000 7f2ca2d9d640 1 -- 172.21.15.38:0/3637256971 learned_addr learned my addr 172.21.15.38:0/3637256971 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.253+0000 7f2cab9f4740 10 client.0 _ll_register_callbacks cb 0x5649c20cca00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:41:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.253+0000 7f2ca2d9d640 1 -- 172.21.15.38:0/3637256971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c2454d40 msgr2=0x5649c245aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.253+0000 7f2ca2d9d640 1 --2- 172.21.15.38:0/3637256971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c2454d40 0x5649c245aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.253+0000 7f2ca2d9d640 1 -- 172.21.15.38:0/3637256971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 msgr2=0x5649c245a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.253+0000 7f2ca2d9d640 1 --2- 172.21.15.38:0/3637256971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 0x5649c245a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.256 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.253+0000 7f2ca2d9d640 1 -- 172.21.15.38:0/3637256971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649c2199620 con 0x5649c245d550 2024-03-26T06:41:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.254+0000 7f2ca2d9d640 1 --2- 172.21.15.38:0/3637256971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c245d550 0x5649c245b020 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c001970 tx=0x7f2c9c0012f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a8b24bf60080154 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.254+0000 7f2c89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-03-26T06:41:39.257 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.254+0000 7f2c89ffb640 1 -- 172.21.15.38:0/3637256971 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c003970 con 0x5649c245d550 2024-03-26T06:41:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.254+0000 7f2c89ffb640 1 -- 172.21.15.38:0/3637256971 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f2c9c003b10 con 0x5649c245d550 2024-03-26T06:41:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.254+0000 7f2cab9f4740 1 -- 172.21.15.38:0/3637256971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5649c23c5b30 con 0x5649c245d550 2024-03-26T06:41:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.255+0000 7f2c89ffb640 1 -- 172.21.15.38:0/3637256971 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c9c003d10 con 0x5649c245d550 2024-03-26T06:41:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.255+0000 7f2cab9f4740 10 client.25274 fetch_fsmap learned FSMap version 29 2024-03-26T06:41:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.255+0000 7f2cab9f4740 1 -- 172.21.15.38:0/3637256971 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5649c246c3c0 con 0x5649c245d550 2024-03-26T06:41:39.258 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.255+0000 7f2c89ffb640 1 -- 172.21.15.38:0/3637256971 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5649c23c5b30 con 0x5649c245d550 2024-03-26T06:41:39.264 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.262+0000 7f2c89ffb640 1 -- 172.21.15.38:0/3637256971 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 29) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f2c9c013610 con 0x5649c245d550 2024-03-26T06:41:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.262+0000 7f2cab9f4740 10 client.25274 fetch_fsmap finished waiting for FSMap version 29 2024-03-26T06:41:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.262+0000 7f2cab9f4740 10 client.25274 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:41:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.262+0000 7f2cab9f4740 20 client.25274 populate_metadata read hostname 'smithi038' 2024-03-26T06:41:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.262+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/3637256971 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5649c246dd50 0x5649c2470210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.262+0000 7f2cab9f4740 4 client.25274 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:41:39.265 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.262+0000 7f2cab9f4740 1 -- 172.21.15.38:0/3637256971 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5649c23bfe80 con 0x5649c246dd50 2024-03-26T06:41:39.266 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.264+0000 7f2ca259c640 1 --2- 172.21.15.38:0/3637256971 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5649c246dd50 0x5649c2470210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.264+0000 7f2ca259c640 1 --2- 172.21.15.38:0/3637256971 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5649c246dd50 0x5649c2470210 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=977335e5bbe4ad61 server_cookie=7092f1c03c30dd37 in_seq=0 out_seq=0 2024-03-26T06:41:39.267 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.264+0000 7f2c89ffb640 10 client.25274 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2c89ffb640 1 -- 172.21.15.38:0/3637256971 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5649c23bfe80 con 0x5649c246dd50 2024-03-26T06:41:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2c89ffb640 10 client.25274 handle_command_reply: tid=0 2024-03-26T06:41:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2cab9f4740 1 client.25274 shutdown 2024-03-26T06:41:39.274 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2cab9f4740 1 -- 172.21.15.38:0/3637256971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c245d550 msgr2=0x5649c245b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/3637256971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c245d550 0x5649c245b020 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c001970 tx=0x7f2c9c0012f0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2cab9f4740 1 -- 172.21.15.38:0/3637256971 shutdown_connections 2024-03-26T06:41:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/3637256971 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5649c246dd50 0x5649c2470210 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/3637256971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c245d550 0x5649c245b020 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/3637256971 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c2454d40 0x5649c245aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/3637256971 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 0x5649c245a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2cab9f4740 1 -- 172.21.15.38:0/3637256971 >> 172.21.15.38:0/3637256971 conn(0x5649c241b5b0 msgr2=0x5649c2450a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2c89ffb640 0 client.25274 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2cab9f4740 1 -- 172.21.15.38:0/3637256971 shutdown_connections 2024-03-26T06:41:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.271+0000 7f2cab9f4740 1 -- 172.21.15.38:0/3637256971 wait complete. 2024-03-26T06:41:39.275 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.272+0000 7f2cab9f4740 20 client.25274 trim_cache size 0 max 16384 2024-03-26T06:41:39.310 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.308+0000 7f2cab9f4740 1 Processor -- start 2024-03-26T06:41:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.308+0000 7f2cab9f4740 1 -- start start 2024-03-26T06:41:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.308+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 0x5649c245ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.308+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2454d40 0x5649c24643f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.308+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 0x5649c2464930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.308+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c84010a00 con 0x5649c2453fd0 2024-03-26T06:41:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.308+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400e570 con 0x5649c2454d40 2024-03-26T06:41:39.311 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.308+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400c6c0 con 0x5649c245d550 2024-03-26T06:41:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.309+0000 7f2ca1d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2454d40 0x5649c24643f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.309+0000 7f2ca1d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2454d40 0x5649c24643f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:55758/0 (socket says 172.21.15.38:55758) 2024-03-26T06:41:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.309+0000 7f2ca1d9b640 1 -- 172.21.15.38:0/979605467 learned_addr learned my addr 172.21.15.38:0/979605467 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.309+0000 7f2ca2d9d640 1 --2- 172.21.15.38:0/979605467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 0x5649c2464930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.309+0000 7f2ca1d9b640 1 -- 172.21.15.38:0/979605467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 msgr2=0x5649c2464930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.309+0000 7f2ca1d9b640 1 --2- 172.21.15.38:0/979605467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 0x5649c2464930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.312 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.309+0000 7f2ca1d9b640 1 -- 172.21.15.38:0/979605467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 msgr2=0x5649c245ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.309+0000 7f2ca1d9b640 1 --2- 172.21.15.38:0/979605467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 0x5649c245ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.309+0000 7f2ca1d9b640 1 -- 172.21.15.38:0/979605467 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5649c24532a0 con 0x5649c2454d40 2024-03-26T06:41:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.309+0000 7f2ca1d9b640 1 --2- 172.21.15.38:0/979605467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2454d40 0x5649c24643f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4068bc0 tx=0x7f2c98009a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37e2dbe83e1c4755 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:39.313 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.310+0000 7f2c8bfff640 1 -- 172.21.15.38:0/979605467 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c98015020 con 0x5649c2454d40 2024-03-26T06:41:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.310+0000 7f2c8bfff640 1 -- 172.21.15.38:0/979605467 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c98011d10 con 0x5649c2454d40 2024-03-26T06:41:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.311+0000 7f2cab9f4740 1 -- 172.21.15.38:0/979605467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2454d40 msgr2=0x5649c24643f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.311+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/979605467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2454d40 0x5649c24643f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ca4068bc0 tx=0x7f2c98009a00 comp rx=0 tx=0).stop 2024-03-26T06:41:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.311+0000 7f2cab9f4740 1 -- 172.21.15.38:0/979605467 shutdown_connections 2024-03-26T06:41:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.311+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/979605467 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 0x5649c2464930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.311+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/979605467 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2454d40 0x5649c24643f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.311+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/979605467 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 0x5649c245ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.311+0000 7f2cab9f4740 1 -- 172.21.15.38:0/979605467 >> 172.21.15.38:0/979605467 conn(0x5649c241b5b0 msgr2=0x5649c244e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:39.314 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.311+0000 7f2cab9f4740 1 -- 172.21.15.38:0/979605467 shutdown_connections 2024-03-26T06:41:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.311+0000 7f2cab9f4740 1 -- 172.21.15.38:0/979605467 wait complete. 2024-03-26T06:41:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.312+0000 7f2cab9f4740 1 -- start start 2024-03-26T06:41:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.312+0000 7f2cab9f4740 1 Processor -- start 2024-03-26T06:41:39.315 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.312+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 0x5649c2464400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.312+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 0x5649c2456890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.312+0000 7f2cab9f4740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2464940 0x5649c2456dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.312+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400cc30 con 0x5649c2453fd0 2024-03-26T06:41:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.312+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400df10 con 0x5649c2464940 2024-03-26T06:41:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.312+0000 7f2cab9f4740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c8400d250 con 0x5649c245d550 2024-03-26T06:41:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.312+0000 7f2cab9f4740 10 client.0 _ll_register_callbacks cb 0x7f2c94004db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:41:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.313+0000 7f2ca2d9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2464940 0x5649c2456dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.313+0000 7f2ca2d9d640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2464940 0x5649c2456dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:55770/0 (socket says 172.21.15.38:55770) 2024-03-26T06:41:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.313+0000 7f2ca2d9d640 1 -- 172.21.15.38:0/4210136904 learned_addr learned my addr 172.21.15.38:0/4210136904 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:39.316 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.313+0000 7f2ca1d9b640 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 0x5649c2456890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.316+0000 7f2ca259c640 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 0x5649c2464400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.316+0000 7f2ca2d9d640 1 -- 172.21.15.38:0/4210136904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 msgr2=0x5649c2456890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.316+0000 7f2ca2d9d640 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 0x5649c2456890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.316+0000 7f2ca2d9d640 1 -- 172.21.15.38:0/4210136904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 msgr2=0x5649c2464400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.316+0000 7f2ca2d9d640 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 0x5649c2464400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.316+0000 7f2ca2d9d640 1 -- 172.21.15.38:0/4210136904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5649c2457410 con 0x5649c2464940 2024-03-26T06:41:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.317+0000 7f2ca2d9d640 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2464940 0x5649c2456dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c00cc90 tx=0x7f2c9c013e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a058276875a84db9 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:39.320 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.317+0000 7f2c89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.121:3300/0 2024-03-26T06:41:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.317+0000 7f2c89ffb640 1 -- 172.21.15.38:0/4210136904 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c013740 con 0x5649c2464940 2024-03-26T06:41:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.317+0000 7f2c89ffb640 1 -- 172.21.15.38:0/4210136904 <== mon.2 v2:172.21.15.121:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f2c9c003eb0 con 0x5649c2464940 2024-03-26T06:41:39.321 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.317+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4210136904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5649c233e710 con 0x5649c2464940 2024-03-26T06:41:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.318+0000 7f2c89ffb640 1 -- 172.21.15.38:0/4210136904 <== mon.2 v2:172.21.15.121:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c9c03d050 con 0x5649c2464940 2024-03-26T06:41:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.318+0000 7f2cab9f4740 10 client.25280 fetch_fsmap learned FSMap version 29 2024-03-26T06:41:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.318+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4210136904 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5649c24c5200 con 0x5649c2464940 2024-03-26T06:41:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.318+0000 7f2c89ffb640 1 -- 172.21.15.38:0/4210136904 <== mon.2 v2:172.21.15.121:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5649c233e710 con 0x5649c2464940 2024-03-26T06:41:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.318+0000 7f2c89ffb640 1 -- 172.21.15.38:0/4210136904 <== mon.2 v2:172.21.15.121:3300/0 5 ==== fsmap(e 29) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f2c9c038030 con 0x5649c2464940 2024-03-26T06:41:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.319+0000 7f2cab9f4740 10 client.25280 fetch_fsmap finished waiting for FSMap version 29 2024-03-26T06:41:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.319+0000 7f2cab9f4740 10 client.25280 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:41:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.319+0000 7f2cab9f4740 20 client.25280 populate_metadata read hostname 'smithi038' 2024-03-26T06:41:39.322 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.319+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5649c24c5840 0x5649c24c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.319+0000 7f2cab9f4740 4 client.25280 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:41:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.319+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4210136904 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5649c238a590 con 0x5649c24c5840 2024-03-26T06:41:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.319+0000 7f2ca259c640 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5649c24c5840 0x5649c24c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.319+0000 7f2c89ffb640 10 client.25280 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:39.323 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.320+0000 7f2ca259c640 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5649c24c5840 0x5649c24c7c60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d53bd0913f0829c6 server_cookie=7e16db4724d2c2c0 in_seq=0 out_seq=0 2024-03-26T06:41:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2c89ffb640 1 -- 172.21.15.38:0/4210136904 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5649c238a590 con 0x5649c24c5840 2024-03-26T06:41:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2c89ffb640 10 client.25280 handle_command_reply: tid=0 2024-03-26T06:41:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2cab9f4740 1 client.25280 shutdown 2024-03-26T06:41:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4210136904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2464940 msgr2=0x5649c2456dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2464940 0x5649c2456dd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c00cc90 tx=0x7f2c9c013e80 comp rx=0 tx=0).stop 2024-03-26T06:41:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4210136904 shutdown_connections 2024-03-26T06:41:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x5649c24c5840 0x5649c24c7c60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x5649c2464940 0x5649c2456dd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x5649c245d550 0x5649c2456890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2cab9f4740 1 --2- 172.21.15.38:0/4210136904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5649c2453fd0 0x5649c2464400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4210136904 >> 172.21.15.38:0/4210136904 conn(0x5649c241b5b0 msgr2=0x5649c2458e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.369+0000 7f2c89ffb640 0 client.25280 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.370+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4210136904 shutdown_connections 2024-03-26T06:41:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.370+0000 7f2cab9f4740 1 -- 172.21.15.38:0/4210136904 wait complete. 2024-03-26T06:41:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.370+0000 7f2cab9f4740 20 client.25280 trim_cache size 0 max 16384 2024-03-26T06:41:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:41:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:41:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:41:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:41:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 -- 172.21.15.38:0/3198841832 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f2c8c0a2c40 msgr2=0x7f2c8c0a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f2c8c0a2c40 0x7f2c8c0a5100 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c001f30 tx=0x7f2c9c001610 comp rx=0 tx=0).stop 2024-03-26T06:41:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 -- 172.21.15.38:0/3198841832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ca4068eb0 msgr2=0x7f2ca4071cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ca4068eb0 0x7f2ca4071cb0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f2c9403d330 tx=0x7f2c940045e0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 -- 172.21.15.38:0/3198841832 shutdown_connections 2024-03-26T06:41:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f2c8c0a2c40 0x7f2c8c0a5100 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2ca41062b0 0x7f2ca410cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f2ca4103d00 0x7f2ca40721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 --2- 172.21.15.38:0/3198841832 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2ca4068eb0 0x7f2ca4071cb0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:39.376 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 -- 172.21.15.38:0/3198841832 >> 172.21.15.38:0/3198841832 conn(0x7f2ca4077040 msgr2=0x7f2ca40ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:39.376 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 -- 172.21.15.38:0/3198841832 shutdown_connections 2024-03-26T06:41:39.376 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:39.371+0000 7f2ca8e12640 1 -- 172.21.15.38:0/3198841832 wait complete. 2024-03-26T06:41:40.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:39 smithi121 ceph-mon[36349]: pgmap v127: 97 pgs: 97 active+clean; 18 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 63 KiB/s rd, 50 MiB/s wr, 9.24k op/s 2024-03-26T06:41:40.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:39 smithi170 ceph-mon[36912]: pgmap v127: 97 pgs: 97 active+clean; 18 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 63 KiB/s rd, 50 MiB/s wr, 9.24k op/s 2024-03-26T06:41:40.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:39 smithi038 ceph-mon[32265]: pgmap v127: 97 pgs: 97 active+clean; 18 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 63 KiB/s rd, 50 MiB/s wr, 9.24k op/s 2024-03-26T06:41:42.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:41 smithi038 ceph-mon[32265]: pgmap v128: 97 pgs: 97 active+clean; 18 GiB data, 79 GiB used, 993 GiB / 1.0 TiB avail; 102 KiB/s rd, 63 MiB/s wr, 11.71k op/s 2024-03-26T06:41:42.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:41 smithi121 ceph-mon[36349]: pgmap v128: 97 pgs: 97 active+clean; 18 GiB data, 79 GiB used, 993 GiB / 1.0 TiB avail; 102 KiB/s rd, 63 MiB/s wr, 11.71k op/s 2024-03-26T06:41:42.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:41 smithi170 ceph-mon[36912]: pgmap v128: 97 pgs: 97 active+clean; 18 GiB data, 79 GiB used, 993 GiB / 1.0 TiB avail; 102 KiB/s rd, 63 MiB/s wr, 11.71k op/s 2024-03-26T06:41:43.296 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:42 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:41:43.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:42 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:41:43.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:42 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:41:44.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:43 smithi038 ceph-mon[32265]: pgmap v129: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 68 KiB/s rd, 49 MiB/s wr, 8.94k op/s 2024-03-26T06:41:44.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:43 smithi121 ceph-mon[36349]: pgmap v129: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 68 KiB/s rd, 49 MiB/s wr, 8.94k op/s 2024-03-26T06:41:44.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:43 smithi170 ceph-mon[36912]: pgmap v129: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 68 KiB/s rd, 49 MiB/s wr, 8.94k op/s 2024-03-26T06:41:45.333 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:44 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:41:45.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:44 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:41:45.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:44 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:41:45.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:44 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:41:45.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:44 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:41:45.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:44 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:41:45.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:44 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:41:45.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:44 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:41:45.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:44 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:41:46.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:46 smithi121 ceph-mon[36349]: pgmap v130: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 68 KiB/s rd, 50 MiB/s wr, 9.07k op/s 2024-03-26T06:41:46.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:46 smithi170 ceph-mon[36912]: pgmap v130: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 68 KiB/s rd, 50 MiB/s wr, 9.07k op/s 2024-03-26T06:41:46.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:46 smithi038 ceph-mon[32265]: pgmap v130: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 68 KiB/s rd, 50 MiB/s wr, 9.07k op/s 2024-03-26T06:41:48.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:48 smithi121 ceph-mon[36349]: pgmap v131: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 75 KiB/s rd, 66 MiB/s wr, 12.59k op/s 2024-03-26T06:41:48.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:48 smithi170 ceph-mon[36912]: pgmap v131: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 75 KiB/s rd, 66 MiB/s wr, 12.59k op/s 2024-03-26T06:41:48.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:48 smithi038 ceph-mon[32265]: pgmap v131: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 75 KiB/s rd, 66 MiB/s wr, 12.59k op/s 2024-03-26T06:41:50.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:50 smithi121 ceph-mon[36349]: pgmap v132: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 48 KiB/s rd, 46 MiB/s wr, 9.17k op/s 2024-03-26T06:41:50.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:50 smithi170 ceph-mon[36912]: pgmap v132: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 48 KiB/s rd, 46 MiB/s wr, 9.17k op/s 2024-03-26T06:41:50.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:50 smithi038 ceph-mon[32265]: pgmap v132: 97 pgs: 97 active+clean; 18 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 48 KiB/s rd, 46 MiB/s wr, 9.17k op/s 2024-03-26T06:41:52.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:52 smithi121 ceph-mon[36349]: pgmap v133: 97 pgs: 97 active+clean; 19 GiB data, 81 GiB used, 991 GiB / 1.0 TiB avail; 86 KiB/s rd, 68 MiB/s wr, 13.57k op/s 2024-03-26T06:41:52.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:52 smithi170 ceph-mon[36912]: pgmap v133: 97 pgs: 97 active+clean; 19 GiB data, 81 GiB used, 991 GiB / 1.0 TiB avail; 86 KiB/s rd, 68 MiB/s wr, 13.57k op/s 2024-03-26T06:41:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.535+0000 7f46ec9df640 1 -- 172.21.15.38:0/3626763438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e810fea0 msgr2=0x7f46e810db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:52.538 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.535+0000 7f46ec9df640 1 --2- 172.21.15.38:0/3626763438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e810fea0 0x7f46e810db90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f46d8009be0 tx=0x7f46d802bf10 comp rx=0 tx=0).stop 2024-03-26T06:41:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.537+0000 7f46ec9df640 1 -- 172.21.15.38:0/3626763438 shutdown_connections 2024-03-26T06:41:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.537+0000 7f46ec9df640 1 --2- 172.21.15.38:0/3626763438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46e811aad0 0x7f46e811cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.537+0000 7f46ec9df640 1 --2- 172.21.15.38:0/3626763438 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e810fea0 0x7f46e810db90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.537+0000 7f46ec9df640 1 --2- 172.21.15.38:0/3626763438 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46e810fa30 0x7f46e810d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.537+0000 7f46ec9df640 1 -- 172.21.15.38:0/3626763438 >> 172.21.15.38:0/3626763438 conn(0x7f46e806c390 msgr2=0x7f46e806c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:52.540 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.537+0000 7f46ec9df640 1 -- 172.21.15.38:0/3626763438 shutdown_connections 2024-03-26T06:41:52.543 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.540+0000 7f46ec9df640 1 -- 172.21.15.38:0/3626763438 wait complete. 2024-03-26T06:41:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.541+0000 7f46ec9df640 1 Processor -- start 2024-03-26T06:41:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.541+0000 7f46ec9df640 1 -- start start 2024-03-26T06:41:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.541+0000 7f46ec9df640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e810fa30 0x7f46e81ac610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.541+0000 7f46ec9df640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46e810fea0 0x7f46e81acb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.541+0000 7f46ec9df640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46e811aad0 0x7f46e81a6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.541+0000 7f46ec9df640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e8120500 con 0x7f46e811aad0 2024-03-26T06:41:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.541+0000 7f46ec9df640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e8120380 con 0x7f46e810fa30 2024-03-26T06:41:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.541+0000 7f46ec9df640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e8120680 con 0x7f46e810fea0 2024-03-26T06:41:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.541+0000 7f46e5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46e810fea0 0x7f46e81acb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.542+0000 7f46e5d9b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46e810fea0 0x7f46e81acb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:45460/0 (socket says 172.21.15.38:45460) 2024-03-26T06:41:52.546 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.542+0000 7f46e5d9b640 1 -- 172.21.15.38:0/3926331227 learned_addr learned my addr 172.21.15.38:0/3926331227 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.542+0000 7f46e659c640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e810fa30 0x7f46e81ac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.542+0000 7f46e6d9d640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46e811aad0 0x7f46e81a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.542+0000 7f46e5d9b640 1 -- 172.21.15.38:0/3926331227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e810fa30 msgr2=0x7f46e81ac610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.542+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e810fa30 0x7f46e81ac610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.542+0000 7f46e5d9b640 1 -- 172.21.15.38:0/3926331227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46e811aad0 msgr2=0x7f46e81a6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.542+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46e811aad0 0x7f46e81a6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.542+0000 7f46e5d9b640 1 -- 172.21.15.38:0/3926331227 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e81a6f50 con 0x7f46e810fea0 2024-03-26T06:41:52.547 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.543+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46e810fea0 0x7f46e81acb50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f46d8033420 tx=0x7f46d8033db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.543+0000 7f46d77fe640 1 -- 172.21.15.38:0/3926331227 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8043020 con 0x7f46e810fea0 2024-03-26T06:41:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.544+0000 7f46d77fe640 1 -- 172.21.15.38:0/3926331227 <== mon.1 v2:172.21.15.170:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d8033480 con 0x7f46e810fea0 2024-03-26T06:41:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.544+0000 7f46ec9df640 1 -- 172.21.15.38:0/3926331227 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e81a71e0 con 0x7f46e810fea0 2024-03-26T06:41:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.544+0000 7f46ec9df640 1 -- 172.21.15.38:0/3926331227 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e81102b0 con 0x7f46e810fea0 2024-03-26T06:41:52.548 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.544+0000 7f46d77fe640 1 -- 172.21.15.38:0/3926331227 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d802cab0 con 0x7f46e810fea0 2024-03-26T06:41:52.551 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.548+0000 7f46d77fe640 1 -- 172.21.15.38:0/3926331227 <== mon.1 v2:172.21.15.170:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f46d8046020 con 0x7f46e810fea0 2024-03-26T06:41:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.549+0000 7f46ef5c3740 1 Processor -- start 2024-03-26T06:41:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.549+0000 7f46ef5c3740 1 -- start start 2024-03-26T06:41:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.550+0000 7f46ef5c3740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed90a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.550+0000 7f46ef5c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed91450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.550+0000 7f46ef5c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed9c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.550+0000 7f46ef5c3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x56236ecf55f0 con 0x56236ed9a550 2024-03-26T06:41:52.554 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.550+0000 7f46ef5c3740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56236ec9e930 con 0x56236ed91d40 2024-03-26T06:41:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.550+0000 7f46ef5c3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56236ecf1690 con 0x56236ed90fd0 2024-03-26T06:41:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.551+0000 7f46e5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed9c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.551+0000 7f46e5d9b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed9c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56550/0 (socket says 172.21.15.38:56550) 2024-03-26T06:41:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.551+0000 7f46e5d9b640 1 -- 172.21.15.38:0/4253973240 learned_addr learned my addr 172.21.15.38:0/4253973240 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:52.555 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.551+0000 7f46e659c640 1 --2- 172.21.15.38:0/4253973240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed91450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.551+0000 7f46e6d9d640 1 --2- 172.21.15.38:0/4253973240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed90a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.552+0000 7f46e5d9b640 1 -- 172.21.15.38:0/4253973240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 msgr2=0x56236ed91450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.552+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/4253973240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed91450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.552+0000 7f46e5d9b640 1 -- 172.21.15.38:0/4253973240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 msgr2=0x56236ed90a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:52.556 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.552+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/4253973240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed90a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.552+0000 7f46e5d9b640 1 -- 172.21.15.38:0/4253973240 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56236ea0a210 con 0x56236ed9a550 2024-03-26T06:41:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.552+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/4253973240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed9c940 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f46d80eed40 tx=0x7f46d80f79d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef3962a2841371fe server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.553+0000 7f46d4ff9640 1 -- 172.21.15.38:0/4253973240 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d80f1dd0 con 0x56236ed9a550 2024-03-26T06:41:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.553+0000 7f46d4ff9640 1 -- 172.21.15.38:0/4253973240 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d80fe070 con 0x56236ed9a550 2024-03-26T06:41:52.557 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.553+0000 7f46d4ff9640 1 -- 172.21.15.38:0/4253973240 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8103b30 con 0x56236ed9a550 2024-03-26T06:41:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.554+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4253973240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 msgr2=0x56236ed9c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.554+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/4253973240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed9c940 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f46d80eed40 tx=0x7f46d80f79d0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.555+0000 7f46d77fe640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f46d00a2bf0 0x7f46d00a50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.558 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.555+0000 7f46d77fe640 1 -- 172.21.15.38:0/3926331227 <== mon.1 v2:172.21.15.170:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f46d80f1610 con 0x7f46e810fea0 2024-03-26T06:41:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.559+0000 7f46e6d9d640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f46d00a2bf0 0x7f46d00a50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.560+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4253973240 shutdown_connections 2024-03-26T06:41:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.560+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/4253973240 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed9c940 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.560+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/4253973240 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed91450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.560+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/4253973240 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed90a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.560+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4253973240 >> 172.21.15.38:0/4253973240 conn(0x56236ed585b0 msgr2=0x56236ed8bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:52.563 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.560+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4253973240 shutdown_connections 2024-03-26T06:41:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.560+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4253973240 wait complete. 2024-03-26T06:41:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.561+0000 7f46ef5c3740 1 -- start start 2024-03-26T06:41:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.561+0000 7f46ef5c3740 1 Processor -- start 2024-03-26T06:41:52.564 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.561+0000 7f46e6d9d640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f46d00a2bf0 0x7f46d00a50b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f46e806a700 tx=0x7f46dc0090e0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.561+0000 7f46ef5c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed97500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.561+0000 7f46ef5c3740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed97a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.561+0000 7f46ef5c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed97f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.561+0000 7f46ef5c3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x56236ecf55f0 con 0x56236ed9a550 2024-03-26T06:41:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.561+0000 7f46ef5c3740 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x56236ecb1d90 con 0x56236ed91d40 2024-03-26T06:41:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.561+0000 7f46ef5c3740 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x56236ed9d040 con 0x56236ed90fd0 2024-03-26T06:41:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.561+0000 7f46ef5c3740 10 client.0 _ll_register_callbacks cb 0x56236ea09a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:41:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.562+0000 7f46e6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed97f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.562+0000 7f46e6d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed97f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:56556/0 (socket says 172.21.15.38:56556) 2024-03-26T06:41:52.565 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.562+0000 7f46e6d9d640 1 -- 172.21.15.38:0/612397015 learned_addr learned my addr 172.21.15.38:0/612397015 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.562+0000 7f46e659c640 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed97500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.562+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed97a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.562+0000 7f46e6d9d640 1 -- 172.21.15.38:0/612397015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 msgr2=0x56236ed97500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.562+0000 7f46e6d9d640 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed97500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.562+0000 7f46e6d9d640 1 -- 172.21.15.38:0/612397015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 msgr2=0x56236ed97a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.562+0000 7f46e6d9d640 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed97a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.562+0000 7f46e6d9d640 1 -- 172.21.15.38:0/612397015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56236ead6620 con 0x56236ed9a550 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.563+0000 7f46e6d9d640 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed97f80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0149c0 tx=0x7f46dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5280e303cd96079 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.563+0000 7f46cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.563+0000 7f46cdffb640 1 -- 172.21.15.38:0/612397015 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46dc025070 con 0x56236ed9a550 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.563+0000 7f46cdffb640 1 -- 172.21.15.38:0/612397015 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f46dc021600 con 0x56236ed9a550 2024-03-26T06:41:52.567 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.563+0000 7f46ef5c3740 1 -- 172.21.15.38:0/612397015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46d80fe070 con 0x56236ed9a550 2024-03-26T06:41:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.564+0000 7f46cdffb640 1 -- 172.21.15.38:0/612397015 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46dc031050 con 0x56236ed9a550 2024-03-26T06:41:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.564+0000 7f46ef5c3740 10 client.15642 fetch_fsmap learned FSMap version 29 2024-03-26T06:41:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.564+0000 7f46ef5c3740 1 -- 172.21.15.38:0/612397015 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56236eda91b0 con 0x56236ed9a550 2024-03-26T06:41:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.564+0000 7f46cdffb640 1 -- 172.21.15.38:0/612397015 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d80fe070 con 0x56236ed9a550 2024-03-26T06:41:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.564+0000 7f46cdffb640 1 -- 172.21.15.38:0/612397015 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 29) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f46dc020070 con 0x56236ed9a550 2024-03-26T06:41:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.564+0000 7f46ef5c3740 10 client.15642 fetch_fsmap finished waiting for FSMap version 29 2024-03-26T06:41:52.568 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.564+0000 7f46ef5c3740 10 client.15642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:41:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.564+0000 7f46ef5c3740 20 client.15642 populate_metadata read hostname 'smithi038' 2024-03-26T06:41:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.565+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56236edaa6e0 0x56236edacb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.565+0000 7f46ef5c3740 4 client.15642 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-26T06:41:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.565+0000 7f46ef5c3740 1 -- 172.21.15.38:0/612397015 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56236ed02b30 con 0x56236edaa6e0 2024-03-26T06:41:52.569 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.565+0000 7f46e659c640 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56236edaa6e0 0x56236edacb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.571+0000 7f46e659c640 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56236edaa6e0 0x56236edacb20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22a1be0546fe2344 server_cookie=9de1ad1f1ed59719 in_seq=0 out_seq=0 2024-03-26T06:41:52.574 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.571+0000 7f46cdffb640 10 client.15642 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.572+0000 7f46cdffb640 1 -- 172.21.15.38:0/612397015 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56236ed02b30 con 0x56236edaa6e0 2024-03-26T06:41:52.575 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.572+0000 7f46cdffb640 10 client.15642 handle_command_reply: tid=0 2024-03-26T06:41:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.575+0000 7f46ef5c3740 1 client.15642 shutdown 2024-03-26T06:41:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.576+0000 7f46ef5c3740 1 -- 172.21.15.38:0/612397015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 msgr2=0x56236ed97f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:52.578 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.576+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed97f80 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f46dc0149c0 tx=0x7f46dc0046c0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.576+0000 7f46ef5c3740 1 -- 172.21.15.38:0/612397015 shutdown_connections 2024-03-26T06:41:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.576+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56236edaa6e0 0x56236edacb20 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.579 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.576+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed97f80 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.576+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed97a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.576+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/612397015 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed97500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.576+0000 7f46ef5c3740 1 -- 172.21.15.38:0/612397015 >> 172.21.15.38:0/612397015 conn(0x56236ed585b0 msgr2=0x56236ed8da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.576+0000 7f46cdffb640 0 client.15642 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.576+0000 7f46ef5c3740 1 -- 172.21.15.38:0/612397015 shutdown_connections 2024-03-26T06:41:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.576+0000 7f46ef5c3740 1 -- 172.21.15.38:0/612397015 wait complete. 2024-03-26T06:41:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.577+0000 7f46ef5c3740 20 client.15642 trim_cache size 0 max 16384 2024-03-26T06:41:52.589 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:52 smithi038 ceph-mon[32265]: pgmap v133: 97 pgs: 97 active+clean; 19 GiB data, 81 GiB used, 991 GiB / 1.0 TiB avail; 86 KiB/s rd, 68 MiB/s wr, 13.57k op/s 2024-03-26T06:41:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.610+0000 7f46ef5c3740 1 Processor -- start 2024-03-26T06:41:52.613 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.610+0000 7f46ef5c3740 1 -- start start 2024-03-26T06:41:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.611+0000 7f46ef5c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed90fd0 0x56236ed99c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.611+0000 7f46ef5c3740 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed95dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.611+0000 7f46ef5c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed9a550 0x56236ed96310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.611+0000 7f46ef5c3740 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c8010b20 con 0x56236ed90fd0 2024-03-26T06:41:52.614 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.611+0000 7f46e5d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed95dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.611+0000 7f46e5d9b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed95dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:57902/0 (socket says 172.21.15.38:57902) 2024-03-26T06:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.611+0000 7f46e5d9b640 1 -- 172.21.15.38:0/4198404993 learned_addr learned my addr 172.21.15.38:0/4198404993 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.611+0000 7f46e6d9d640 1 --2- 172.21.15.38:0/4198404993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed9a550 0x56236ed96310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.611+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4198404993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c800e690 con 0x56236ed91d40 2024-03-26T06:41:52.615 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.612+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4198404993 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c800c730 con 0x56236ed9a550 2024-03-26T06:41:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.612+0000 7f46e5d9b640 1 -- 172.21.15.38:0/4198404993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed9a550 msgr2=0x56236ed96310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.612+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/4198404993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed9a550 0x56236ed96310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.612+0000 7f46e5d9b640 1 -- 172.21.15.38:0/4198404993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed90fd0 msgr2=0x56236ed99c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.612+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/4198404993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed90fd0 0x56236ed99c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.612+0000 7f46e5d9b640 1 -- 172.21.15.38:0/4198404993 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56236ed8be80 con 0x56236ed91d40 2024-03-26T06:41:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.612+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/4198404993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed95dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46d80ba6e0 tx=0x7f46d802bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=272980680eb50188 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.613+0000 7f46cffff640 1 -- 172.21.15.38:0/4198404993 <== mon.2 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d80b9fb0 con 0x56236ed91d40 2024-03-26T06:41:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.613+0000 7f46cffff640 1 -- 172.21.15.38:0/4198404993 <== mon.2 v2:172.21.15.121:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46d80fe040 con 0x56236ed91d40 2024-03-26T06:41:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.614+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4198404993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 msgr2=0x56236ed95dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.614+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/4198404993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed95dd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f46d80ba6e0 tx=0x7f46d802bf10 comp rx=0 tx=0).stop 2024-03-26T06:41:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.614+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4198404993 shutdown_connections 2024-03-26T06:41:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.614+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/4198404993 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed9a550 0x56236ed96310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.614+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/4198404993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236ed91d40 0x56236ed95dd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.614+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/4198404993 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed90fd0 0x56236ed99c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.614+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4198404993 >> 172.21.15.38:0/4198404993 conn(0x56236ed585b0 msgr2=0x56236edb1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.615+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4198404993 shutdown_connections 2024-03-26T06:41:52.618 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.615+0000 7f46ef5c3740 1 -- 172.21.15.38:0/4198404993 wait complete. 2024-03-26T06:41:52.619 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.615+0000 7f46ef5c3740 1 -- start start 2024-03-26T06:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.615+0000 7f46ef5c3740 1 Processor -- start 2024-03-26T06:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.616+0000 7f46ef5c3740 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed9f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.616+0000 7f46ef5c3740 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed9f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.616+0000 7f46e659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed9f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.616+0000 7f46e659c640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed9f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.170:3300/0 says I am v2:172.21.15.38:45504/0 (socket says 172.21.15.38:45504) 2024-03-26T06:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.616+0000 7f46e659c640 1 -- 172.21.15.38:0/802594863 learned_addr learned my addr 172.21.15.38:0/802594863 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.617+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236edf9bd0 0x56236edfa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.617+0000 7f46ef5c3740 1 -- 172.21.15.38:0/802594863 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c800cd30 con 0x56236ed9a550 2024-03-26T06:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.617+0000 7f46ef5c3740 1 -- 172.21.15.38:0/802594863 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c800e030 con 0x56236edf9bd0 2024-03-26T06:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.617+0000 7f46ef5c3740 1 -- 172.21.15.38:0/802594863 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c800d370 con 0x56236ed90fd0 2024-03-26T06:41:52.620 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.617+0000 7f46e6d9d640 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236edf9bd0 0x56236edfa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.617+0000 7f46ef5c3740 10 client.0 _ll_register_callbacks cb 0x7f46d80f4f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-26T06:41:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.617+0000 7f46e659c640 1 -- 172.21.15.38:0/802594863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236edf9bd0 msgr2=0x56236edfa050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.617+0000 7f46e659c640 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236edf9bd0 0x56236edfa050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.617+0000 7f46e659c640 1 -- 172.21.15.38:0/802594863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 msgr2=0x56236ed9f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-26T06:41:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.617+0000 7f46e659c640 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed9f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.621 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.617+0000 7f46e659c640 1 -- 172.21.15.38:0/802594863 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56236edfa590 con 0x56236ed90fd0 2024-03-26T06:41:52.622 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.619+0000 7f46e659c640 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed9f490 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f46e000ba20 tx=0x7f46e000be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=230963b16ebb0b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:41:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.619+0000 7f46cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.170:3300/0 2024-03-26T06:41:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.620+0000 7f46cdffb640 1 -- 172.21.15.38:0/802594863 <== mon.1 v2:172.21.15.170:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e001e070 con 0x56236ed90fd0 2024-03-26T06:41:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.620+0000 7f46ef5c3740 1 -- 172.21.15.38:0/802594863 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56236ecfce80 con 0x56236ed90fd0 2024-03-26T06:41:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.620+0000 7f46cdffb640 1 -- 172.21.15.38:0/802594863 <== mon.1 v2:172.21.15.170:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8541+0+0 (secure 0 0 0) 0x7f46e001e210 con 0x56236ed90fd0 2024-03-26T06:41:52.623 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.620+0000 7f46cdffb640 1 -- 172.21.15.38:0/802594863 <== mon.1 v2:172.21.15.170:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46e001e410 con 0x56236ed90fd0 2024-03-26T06:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.620+0000 7f46ef5c3740 10 client.25186 fetch_fsmap learned FSMap version 29 2024-03-26T06:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.620+0000 7f46ef5c3740 1 -- 172.21.15.38:0/802594863 --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56236ee01a00 con 0x56236ed90fd0 2024-03-26T06:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.621+0000 7f46cdffb640 1 -- 172.21.15.38:0/802594863 <== mon.1 v2:172.21.15.170:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56236ecfce80 con 0x56236ed90fd0 2024-03-26T06:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.621+0000 7f46cdffb640 1 -- 172.21.15.38:0/802594863 <== mon.1 v2:172.21.15.170:3300/0 5 ==== fsmap(e 29) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f46e0011c80 con 0x56236ed90fd0 2024-03-26T06:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.621+0000 7f46ef5c3740 10 client.25186 fetch_fsmap finished waiting for FSMap version 29 2024-03-26T06:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.621+0000 7f46ef5c3740 10 client.25186 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-26T06:41:52.624 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.621+0000 7f46ef5c3740 20 client.25186 populate_metadata read hostname 'smithi038' 2024-03-26T06:41:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.621+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56236ee021c0 0x56236ee045e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:41:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.621+0000 7f46ef5c3740 4 client.25186 mds_command: new command op to 24539 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-26T06:41:52.625 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.621+0000 7f46ef5c3740 1 -- 172.21.15.38:0/802594863 --> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56236ec7b710 con 0x56236ee021c0 2024-03-26T06:41:52.627 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.624+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56236ee021c0 0x56236ee045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:41:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.625+0000 7f46cdffb640 10 client.25186 ms_handle_connect on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.625+0000 7f46e5d9b640 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56236ee021c0 0x56236ee045e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd1ab78095a38194 server_cookie=ff0165df6d8e32c in_seq=0 out_seq=0 2024-03-26T06:41:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46cdffb640 1 -- 172.21.15.38:0/802594863 <== mds.0 v2:172.21.15.121:6838/359437185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56236ec7b710 con 0x56236ee021c0 2024-03-26T06:41:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46cdffb640 10 client.25186 handle_command_reply: tid=0 2024-03-26T06:41:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46ef5c3740 1 client.25186 shutdown 2024-03-26T06:41:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46ef5c3740 1 -- 172.21.15.38:0/802594863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 msgr2=0x56236ed9f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:52.898 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed9f490 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f46e000ba20 tx=0x7f46e000be20 comp rx=0 tx=0).stop 2024-03-26T06:41:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46ef5c3740 1 -- 172.21.15.38:0/802594863 shutdown_connections 2024-03-26T06:41:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.121:6838/359437185,v1:172.21.15.121:6839/359437185] conn(0x56236ee021c0 0x56236ee045e0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x56236edf9bd0 0x56236edfa050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56236ed9a550 0x56236ed9f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46ef5c3740 1 --2- 172.21.15.38:0/802594863 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x56236ed90fd0 0x56236ed9f490 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.899 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46ef5c3740 1 -- 172.21.15.38:0/802594863 >> 172.21.15.38:0/802594863 conn(0x56236ed585b0 msgr2=0x56236ed95d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46cdffb640 0 client.25186 ms_handle_reset on v2:172.21.15.121:6838/359437185 2024-03-26T06:41:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46ef5c3740 1 -- 172.21.15.38:0/802594863 shutdown_connections 2024-03-26T06:41:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.895+0000 7f46ef5c3740 1 -- 172.21.15.38:0/802594863 wait complete. 2024-03-26T06:41:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.896+0000 7f46ef5c3740 20 client.25186 trim_cache size 0 max 16384 2024-03-26T06:41:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:{ 2024-03-26T06:41:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "message": "", 2024-03-26T06:41:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout: "return_code": 0 2024-03-26T06:41:52.900 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stdout:} 2024-03-26T06:41:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.901+0000 7f46ec9df640 1 -- 172.21.15.38:0/3926331227 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f46d00a2bf0 msgr2=0x7f46d00a50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.901+0000 7f46ec9df640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f46d00a2bf0 0x7f46d00a50b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f46e806a700 tx=0x7f46dc0090e0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.901+0000 7f46ec9df640 1 -- 172.21.15.38:0/3926331227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46e810fea0 msgr2=0x7f46e81acb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:41:52.904 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.901+0000 7f46ec9df640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46e810fea0 0x7f46e81acb50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f46d8033420 tx=0x7f46d8033db0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.906+0000 7f46ec9df640 1 -- 172.21.15.38:0/3926331227 shutdown_connections 2024-03-26T06:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.906+0000 7f46ec9df640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f46d00a2bf0 0x7f46d00a50b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.906+0000 7f46ec9df640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46e811aad0 0x7f46e81a6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.906+0000 7f46ec9df640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f46e810fea0 0x7f46e81acb50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.906+0000 7f46ec9df640 1 --2- 172.21.15.38:0/3926331227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f46e810fa30 0x7f46e81ac610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:41:52.909 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.906+0000 7f46ec9df640 1 -- 172.21.15.38:0/3926331227 >> 172.21.15.38:0/3926331227 conn(0x7f46e806c390 msgr2=0x7f46e8071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:41:52.913 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.910+0000 7f46ec9df640 1 -- 172.21.15.38:0/3926331227 shutdown_connections 2024-03-26T06:41:52.915 INFO:teuthology.task.background_exec.ubuntu@smithi038.front.sepia.ceph.com.smithi038.stderr:2024-03-26T06:41:52.913+0000 7f46ec9df640 1 -- 172.21.15.38:0/3926331227 wait complete. 2024-03-26T06:41:54.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:54 smithi170 ceph-mon[36912]: pgmap v134: 97 pgs: 97 active+clean; 19 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 47 KiB/s rd, 68 MiB/s wr, 13.84k op/s 2024-03-26T06:41:54.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:54 smithi038 ceph-mon[32265]: pgmap v134: 97 pgs: 97 active+clean; 19 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 47 KiB/s rd, 68 MiB/s wr, 13.84k op/s 2024-03-26T06:41:54.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:54 smithi121 ceph-mon[36349]: pgmap v134: 97 pgs: 97 active+clean; 19 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 47 KiB/s rd, 68 MiB/s wr, 13.84k op/s 2024-03-26T06:41:56.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:56 smithi038 ceph-mon[32265]: pgmap v135: 97 pgs: 97 active+clean; 19 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 47 KiB/s rd, 63 MiB/s wr, 12.94k op/s 2024-03-26T06:41:56.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:56 smithi121 ceph-mon[36349]: pgmap v135: 97 pgs: 97 active+clean; 19 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 47 KiB/s rd, 63 MiB/s wr, 12.94k op/s 2024-03-26T06:41:56.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:56 smithi170 ceph-mon[36912]: pgmap v135: 97 pgs: 97 active+clean; 19 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 47 KiB/s rd, 63 MiB/s wr, 12.94k op/s 2024-03-26T06:41:58.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:58 smithi038 ceph-mon[32265]: pgmap v136: 97 pgs: 97 active+clean; 19 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 135 KiB/s rd, 95 MiB/s wr, 19.06k op/s 2024-03-26T06:41:58.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:41:58 smithi038 ceph-mon[32265]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:41:58.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:58 smithi121 ceph-mon[36349]: pgmap v136: 97 pgs: 97 active+clean; 19 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 135 KiB/s rd, 95 MiB/s wr, 19.06k op/s 2024-03-26T06:41:58.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:41:58 smithi121 ceph-mon[36349]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:41:58.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:58 smithi170 ceph-mon[36912]: pgmap v136: 97 pgs: 97 active+clean; 19 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 135 KiB/s rd, 95 MiB/s wr, 19.06k op/s 2024-03-26T06:41:58.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:41:58 smithi170 ceph-mon[36912]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-03-26T06:42:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:00 smithi038 ceph-mon[32265]: pgmap v137: 97 pgs: 97 active+clean; 19 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 127 KiB/s rd, 72 MiB/s wr, 14.25k op/s 2024-03-26T06:42:00.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:00 smithi121 ceph-mon[36349]: pgmap v137: 97 pgs: 97 active+clean; 19 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 127 KiB/s rd, 72 MiB/s wr, 14.25k op/s 2024-03-26T06:42:00.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:00 smithi170 ceph-mon[36912]: pgmap v137: 97 pgs: 97 active+clean; 19 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 127 KiB/s rd, 72 MiB/s wr, 14.25k op/s 2024-03-26T06:42:02.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:02 smithi038 ceph-mon[32265]: pgmap v138: 97 pgs: 97 active+clean; 19 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 128 KiB/s rd, 83 MiB/s wr, 16.36k op/s 2024-03-26T06:42:02.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:02 smithi121 ceph-mon[36349]: pgmap v138: 97 pgs: 97 active+clean; 19 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 128 KiB/s rd, 83 MiB/s wr, 16.36k op/s 2024-03-26T06:42:02.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:02 smithi170 ceph-mon[36912]: pgmap v138: 97 pgs: 97 active+clean; 19 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 128 KiB/s rd, 83 MiB/s wr, 16.36k op/s 2024-03-26T06:42:04.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:04 smithi038 ceph-mon[32265]: pgmap v139: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 226 KiB/s rd, 93 MiB/s wr, 17.50k op/s 2024-03-26T06:42:04.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:04 smithi121 ceph-mon[36349]: pgmap v139: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 226 KiB/s rd, 93 MiB/s wr, 17.50k op/s 2024-03-26T06:42:04.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:04 smithi170 ceph-mon[36912]: pgmap v139: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 226 KiB/s rd, 93 MiB/s wr, 17.50k op/s 2024-03-26T06:42:04.986 INFO:tasks.workunit.client.0.smithi038.stdout:Wrote -1 instead of 4096 bytes. 2024-03-26T06:42:04.986 INFO:tasks.workunit.client.0.smithi038.stdout:Probably out of disk space 2024-03-26T06:42:04.986 INFO:tasks.workunit.client.0.smithi038.stderr:write: Resource temporarily unavailable 2024-03-26T06:42:05.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:42:05.041 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-03-26T06:42:05.041 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-26T06:42:05.401 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7e98f5f6b535defc7fa715b456904ffc0500aa9f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_7e98f5f6b535defc7fa715b456904ffc0500aa9f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7e98f5f6b535defc7fa715b456904ffc0500aa9f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi038 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7e98f5f6b535defc7fa715b456904ffc0500aa9f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-03-26T06:42:05.645 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d6753d2be5844c8cba3426e9c8aa116e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7e98f5f6b535defc7fa715b456904ffc0500aa9f/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_7e98f5f6b535defc7fa715b456904ffc0500aa9f/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7e98f5f6b535defc7fa715b456904ffc0500aa9f/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi038 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7e98f5f6b535defc7fa715b456904ffc0500aa9f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-03-26T06:42:05.648 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-26T06:42:05.657 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T06:42:05.670 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-26T06:42:05.683 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-26T06:42:05.684 INFO:teuthology.task.exec:Executing custom commands... 2024-03-26T06:42:05.684 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T06:42:05.684 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-26T06:42:05.741 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-26T06:42:06.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:06 smithi038 ceph-mon[32265]: pgmap v140: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 226 KiB/s rd, 79 MiB/s wr, 14.75k op/s 2024-03-26T06:42:06.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:06 smithi121 ceph-mon[36349]: pgmap v140: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 226 KiB/s rd, 79 MiB/s wr, 14.75k op/s 2024-03-26T06:42:06.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:06 smithi170 ceph-mon[36912]: pgmap v140: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 226 KiB/s rd, 79 MiB/s wr, 14.75k op/s 2024-03-26T06:42:08.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:08 smithi038 ceph-mon[32265]: pgmap v141: 97 pgs: 97 active+clean; 20 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 329 KiB/s rd, 111 MiB/s wr, 20.42k op/s 2024-03-26T06:42:08.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:08 smithi121 ceph-mon[36349]: pgmap v141: 97 pgs: 97 active+clean; 20 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 329 KiB/s rd, 111 MiB/s wr, 20.42k op/s 2024-03-26T06:42:08.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:08 smithi170 ceph-mon[36912]: pgmap v141: 97 pgs: 97 active+clean; 20 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 329 KiB/s rd, 111 MiB/s wr, 20.42k op/s 2024-03-26T06:42:10.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:10 smithi038 ceph-mon[32265]: pgmap v142: 97 pgs: 97 active+clean; 20 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 240 KiB/s rd, 75 MiB/s wr, 13.31k op/s 2024-03-26T06:42:10.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:10 smithi121 ceph-mon[36349]: pgmap v142: 97 pgs: 97 active+clean; 20 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 240 KiB/s rd, 75 MiB/s wr, 13.31k op/s 2024-03-26T06:42:10.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:10 smithi170 ceph-mon[36912]: pgmap v142: 97 pgs: 97 active+clean; 20 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 240 KiB/s rd, 75 MiB/s wr, 13.31k op/s 2024-03-26T06:42:12.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:12 smithi038 ceph-mon[32265]: pgmap v143: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 241 KiB/s rd, 81 MiB/s wr, 14.35k op/s 2024-03-26T06:42:12.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:12 smithi121 ceph-mon[36349]: pgmap v143: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 241 KiB/s rd, 81 MiB/s wr, 14.35k op/s 2024-03-26T06:42:12.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:12 smithi170 ceph-mon[36912]: pgmap v143: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 241 KiB/s rd, 81 MiB/s wr, 14.35k op/s 2024-03-26T06:42:14.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:14 smithi038 ceph-mon[32265]: pgmap v144: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 248 KiB/s rd, 83 MiB/s wr, 14.53k op/s 2024-03-26T06:42:14.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:14 smithi121 ceph-mon[36349]: pgmap v144: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 248 KiB/s rd, 83 MiB/s wr, 14.53k op/s 2024-03-26T06:42:14.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:14 smithi170 ceph-mon[36912]: pgmap v144: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 248 KiB/s rd, 83 MiB/s wr, 14.53k op/s 2024-03-26T06:42:16.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:16 smithi038 ceph-mon[32265]: pgmap v145: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 111 KiB/s rd, 51 MiB/s wr, 8.99k op/s 2024-03-26T06:42:16.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:16 smithi121 ceph-mon[36349]: pgmap v145: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 111 KiB/s rd, 51 MiB/s wr, 8.99k op/s 2024-03-26T06:42:16.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:16 smithi170 ceph-mon[36912]: pgmap v145: 97 pgs: 97 active+clean; 20 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 111 KiB/s rd, 51 MiB/s wr, 8.99k op/s 2024-03-26T06:42:18.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:18 smithi038 ceph-mon[32265]: pgmap v146: 97 pgs: 97 active+clean; 20 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 129 KiB/s rd, 68 MiB/s wr, 11.90k op/s 2024-03-26T06:42:18.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:18 smithi121 ceph-mon[36349]: pgmap v146: 97 pgs: 97 active+clean; 20 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 129 KiB/s rd, 68 MiB/s wr, 11.90k op/s 2024-03-26T06:42:18.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:18 smithi170 ceph-mon[36912]: pgmap v146: 97 pgs: 97 active+clean; 20 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 129 KiB/s rd, 68 MiB/s wr, 11.90k op/s 2024-03-26T06:42:20.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:20 smithi038 ceph-mon[32265]: pgmap v147: 97 pgs: 97 active+clean; 20 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 26 KiB/s rd, 36 MiB/s wr, 6.24k op/s 2024-03-26T06:42:20.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:20 smithi121 ceph-mon[36349]: pgmap v147: 97 pgs: 97 active+clean; 20 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 26 KiB/s rd, 36 MiB/s wr, 6.24k op/s 2024-03-26T06:42:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:20 smithi170 ceph-mon[36912]: pgmap v147: 97 pgs: 97 active+clean; 20 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 26 KiB/s rd, 36 MiB/s wr, 6.24k op/s 2024-03-26T06:42:22.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:22 smithi038 ceph-mon[32265]: pgmap v148: 97 pgs: 97 active+clean; 20 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 26 KiB/s rd, 43 MiB/s wr, 7.61k op/s 2024-03-26T06:42:22.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:22 smithi121 ceph-mon[36349]: pgmap v148: 97 pgs: 97 active+clean; 20 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 26 KiB/s rd, 43 MiB/s wr, 7.61k op/s 2024-03-26T06:42:22.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:22 smithi170 ceph-mon[36912]: pgmap v148: 97 pgs: 97 active+clean; 20 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 26 KiB/s rd, 43 MiB/s wr, 7.61k op/s 2024-03-26T06:42:24.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:24 smithi038 ceph-mon[32265]: pgmap v149: 97 pgs: 97 active+clean; 21 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 37 KiB/s rd, 55 MiB/s wr, 9.84k op/s 2024-03-26T06:42:24.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:24 smithi121 ceph-mon[36349]: pgmap v149: 97 pgs: 97 active+clean; 21 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 37 KiB/s rd, 55 MiB/s wr, 9.84k op/s 2024-03-26T06:42:24.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:24 smithi170 ceph-mon[36912]: pgmap v149: 97 pgs: 97 active+clean; 21 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 37 KiB/s rd, 55 MiB/s wr, 9.84k op/s 2024-03-26T06:42:26.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:26 smithi038 ceph-mon[32265]: pgmap v150: 97 pgs: 97 active+clean; 21 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 30 KiB/s rd, 42 MiB/s wr, 7.56k op/s 2024-03-26T06:42:26.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:26 smithi121 ceph-mon[36349]: pgmap v150: 97 pgs: 97 active+clean; 21 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 30 KiB/s rd, 42 MiB/s wr, 7.56k op/s 2024-03-26T06:42:26.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:26 smithi170 ceph-mon[36912]: pgmap v150: 97 pgs: 97 active+clean; 21 GiB data, 85 GiB used, 988 GiB / 1.0 TiB avail; 30 KiB/s rd, 42 MiB/s wr, 7.56k op/s 2024-03-26T06:42:28.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:28 smithi038 ceph-mon[32265]: pgmap v151: 97 pgs: 97 active+clean; 21 GiB data, 86 GiB used, 987 GiB / 1.0 TiB avail; 47 KiB/s rd, 61 MiB/s wr, 11.02k op/s 2024-03-26T06:42:28.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:28 smithi121 ceph-mon[36349]: pgmap v151: 97 pgs: 97 active+clean; 21 GiB data, 86 GiB used, 987 GiB / 1.0 TiB avail; 47 KiB/s rd, 61 MiB/s wr, 11.02k op/s 2024-03-26T06:42:28.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:28 smithi170 ceph-mon[36912]: pgmap v151: 97 pgs: 97 active+clean; 21 GiB data, 86 GiB used, 987 GiB / 1.0 TiB avail; 47 KiB/s rd, 61 MiB/s wr, 11.02k op/s 2024-03-26T06:42:30.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:30 smithi038 ceph-mon[32265]: pgmap v152: 97 pgs: 97 active+clean; 21 GiB data, 86 GiB used, 987 GiB / 1.0 TiB avail; 29 KiB/s rd, 44 MiB/s wr, 8.10k op/s 2024-03-26T06:42:30.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:30 smithi121 ceph-mon[36349]: pgmap v152: 97 pgs: 97 active+clean; 21 GiB data, 86 GiB used, 987 GiB / 1.0 TiB avail; 29 KiB/s rd, 44 MiB/s wr, 8.10k op/s 2024-03-26T06:42:30.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:30 smithi170 ceph-mon[36912]: pgmap v152: 97 pgs: 97 active+clean; 21 GiB data, 86 GiB used, 987 GiB / 1.0 TiB avail; 29 KiB/s rd, 44 MiB/s wr, 8.10k op/s 2024-03-26T06:42:32.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:32 smithi038 ceph-mon[32265]: pgmap v153: 97 pgs: 97 active+clean; 21 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 29 KiB/s rd, 48 MiB/s wr, 8.78k op/s 2024-03-26T06:42:32.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:32 smithi121 ceph-mon[36349]: pgmap v153: 97 pgs: 97 active+clean; 21 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 29 KiB/s rd, 48 MiB/s wr, 8.78k op/s 2024-03-26T06:42:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:32 smithi170 ceph-mon[36912]: pgmap v153: 97 pgs: 97 active+clean; 21 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 29 KiB/s rd, 48 MiB/s wr, 8.78k op/s 2024-03-26T06:42:34.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:34 smithi038 ceph-mon[32265]: pgmap v154: 97 pgs: 97 active+clean; 21 GiB data, 83 GiB used, 989 GiB / 1.0 TiB avail; 41 KiB/s rd, 53 MiB/s wr, 9.59k op/s 2024-03-26T06:42:34.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:34 smithi121 ceph-mon[36349]: pgmap v154: 97 pgs: 97 active+clean; 21 GiB data, 83 GiB used, 989 GiB / 1.0 TiB avail; 41 KiB/s rd, 53 MiB/s wr, 9.59k op/s 2024-03-26T06:42:34.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:34 smithi170 ceph-mon[36912]: pgmap v154: 97 pgs: 97 active+clean; 21 GiB data, 83 GiB used, 989 GiB / 1.0 TiB avail; 41 KiB/s rd, 53 MiB/s wr, 9.59k op/s 2024-03-26T06:42:36.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:36 smithi038 ceph-mon[32265]: pgmap v155: 97 pgs: 97 active+clean; 21 GiB data, 83 GiB used, 989 GiB / 1.0 TiB avail; 29 KiB/s rd, 35 MiB/s wr, 6.31k op/s 2024-03-26T06:42:36.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:36 smithi121 ceph-mon[36349]: pgmap v155: 97 pgs: 97 active+clean; 21 GiB data, 83 GiB used, 989 GiB / 1.0 TiB avail; 29 KiB/s rd, 35 MiB/s wr, 6.31k op/s 2024-03-26T06:42:36.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:36 smithi170 ceph-mon[36912]: pgmap v155: 97 pgs: 97 active+clean; 21 GiB data, 83 GiB used, 989 GiB / 1.0 TiB avail; 29 KiB/s rd, 35 MiB/s wr, 6.31k op/s 2024-03-26T06:42:38.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:38 smithi038 ceph-mon[32265]: pgmap v156: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 39 KiB/s rd, 51 MiB/s wr, 9.14k op/s 2024-03-26T06:42:38.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:38 smithi121 ceph-mon[36349]: pgmap v156: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 39 KiB/s rd, 51 MiB/s wr, 9.14k op/s 2024-03-26T06:42:38.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:38 smithi170 ceph-mon[36912]: pgmap v156: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 39 KiB/s rd, 51 MiB/s wr, 9.14k op/s 2024-03-26T06:42:40.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:40 smithi038 ceph-mon[32265]: pgmap v157: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 22 KiB/s rd, 32 MiB/s wr, 5.69k op/s 2024-03-26T06:42:40.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:40 smithi121 ceph-mon[36349]: pgmap v157: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 22 KiB/s rd, 32 MiB/s wr, 5.69k op/s 2024-03-26T06:42:40.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:40 smithi170 ceph-mon[36912]: pgmap v157: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 22 KiB/s rd, 32 MiB/s wr, 5.69k op/s 2024-03-26T06:42:42.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:42 smithi038 ceph-mon[32265]: pgmap v158: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 23 KiB/s rd, 33 MiB/s wr, 5.86k op/s 2024-03-26T06:42:42.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:42 smithi121 ceph-mon[36349]: pgmap v158: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 23 KiB/s rd, 33 MiB/s wr, 5.86k op/s 2024-03-26T06:42:42.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:42 smithi170 ceph-mon[36912]: pgmap v158: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 23 KiB/s rd, 33 MiB/s wr, 5.86k op/s 2024-03-26T06:42:44.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:44 smithi121 ceph-mon[36349]: pgmap v159: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 30 KiB/s rd, 29 MiB/s wr, 5.25k op/s 2024-03-26T06:42:44.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:44 smithi170 ceph-mon[36912]: pgmap v159: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 30 KiB/s rd, 29 MiB/s wr, 5.25k op/s 2024-03-26T06:42:44.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:44 smithi038 ceph-mon[32265]: pgmap v159: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 30 KiB/s rd, 29 MiB/s wr, 5.25k op/s 2024-03-26T06:42:45.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:45 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:42:45.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:45 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:42:45.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:45 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:42:45.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:45 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:42:45.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:45 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:42:45.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:45 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:42:45.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:45 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:42:45.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:45 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:42:45.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:45 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:42:45.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:45 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:42:45.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:45 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:42:45.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:45 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:42:46.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:46 smithi121 ceph-mon[36349]: pgmap v160: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 3.08k op/s 2024-03-26T06:42:46.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:46 smithi170 ceph-mon[36912]: pgmap v160: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 3.08k op/s 2024-03-26T06:42:46.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:46 smithi038 ceph-mon[32265]: pgmap v160: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 19 KiB/s rd, 17 MiB/s wr, 3.08k op/s 2024-03-26T06:42:48.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:48 smithi121 ceph-mon[36349]: pgmap v161: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 24 KiB/s rd, 17 MiB/s wr, 3.08k op/s 2024-03-26T06:42:48.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:48 smithi170 ceph-mon[36912]: pgmap v161: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 24 KiB/s rd, 17 MiB/s wr, 3.08k op/s 2024-03-26T06:42:48.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:48 smithi038 ceph-mon[32265]: pgmap v161: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 24 KiB/s rd, 17 MiB/s wr, 3.08k op/s 2024-03-26T06:42:50.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:50 smithi121 ceph-mon[36349]: pgmap v162: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.3 MiB/s wr, 243 op/s 2024-03-26T06:42:50.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:50 smithi170 ceph-mon[36912]: pgmap v162: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.3 MiB/s wr, 243 op/s 2024-03-26T06:42:50.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:50 smithi038 ceph-mon[32265]: pgmap v162: 97 pgs: 97 active+clean; 21 GiB data, 84 GiB used, 988 GiB / 1.0 TiB avail; 14 KiB/s rd, 1.3 MiB/s wr, 243 op/s 2024-03-26T06:42:52.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:52 smithi121 ceph-mon[36349]: pgmap v163: 97 pgs: 97 active+clean; 21 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 15 KiB/s rd, 6.4 MiB/s wr, 1.02k op/s 2024-03-26T06:42:52.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:52 smithi170 ceph-mon[36912]: pgmap v163: 97 pgs: 97 active+clean; 21 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 15 KiB/s rd, 6.4 MiB/s wr, 1.02k op/s 2024-03-26T06:42:52.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:52 smithi038 ceph-mon[32265]: pgmap v163: 97 pgs: 97 active+clean; 21 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 15 KiB/s rd, 6.4 MiB/s wr, 1.02k op/s 2024-03-26T06:42:54.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:54 smithi121 ceph-mon[36349]: pgmap v164: 97 pgs: 97 active+clean; 21 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 2.34k op/s 2024-03-26T06:42:54.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:54 smithi170 ceph-mon[36912]: pgmap v164: 97 pgs: 97 active+clean; 21 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 2.34k op/s 2024-03-26T06:42:54.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:54 smithi038 ceph-mon[32265]: pgmap v164: 97 pgs: 97 active+clean; 21 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 17 KiB/s rd, 15 MiB/s wr, 2.34k op/s 2024-03-26T06:42:56.636 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:56 smithi038 ceph-mon[32265]: pgmap v165: 97 pgs: 97 active+clean; 21 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 8.8 KiB/s rd, 15 MiB/s wr, 2.27k op/s 2024-03-26T06:42:56.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:56 smithi121 ceph-mon[36349]: pgmap v165: 97 pgs: 97 active+clean; 21 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 8.8 KiB/s rd, 15 MiB/s wr, 2.27k op/s 2024-03-26T06:42:56.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:56 smithi170 ceph-mon[36912]: pgmap v165: 97 pgs: 97 active+clean; 21 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 8.8 KiB/s rd, 15 MiB/s wr, 2.27k op/s 2024-03-26T06:42:58.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:42:58 smithi121 ceph-mon[36349]: pgmap v166: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 2.40k op/s 2024-03-26T06:42:58.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:42:58 smithi170 ceph-mon[36912]: pgmap v166: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 2.40k op/s 2024-03-26T06:42:58.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:42:58 smithi038 ceph-mon[32265]: pgmap v166: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 16 KiB/s rd, 15 MiB/s wr, 2.40k op/s 2024-03-26T06:43:00.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:00 smithi170 ceph-mon[36912]: pgmap v167: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 10 KiB/s rd, 15 MiB/s wr, 2.39k op/s 2024-03-26T06:43:00.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:00 smithi038 ceph-mon[32265]: pgmap v167: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 10 KiB/s rd, 15 MiB/s wr, 2.39k op/s 2024-03-26T06:43:00.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:00 smithi121 ceph-mon[36349]: pgmap v167: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 10 KiB/s rd, 15 MiB/s wr, 2.39k op/s 2024-03-26T06:43:02.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:02 smithi170 ceph-mon[36912]: pgmap v168: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 3.16k op/s 2024-03-26T06:43:02.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:02 smithi038 ceph-mon[32265]: pgmap v168: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 3.16k op/s 2024-03-26T06:43:02.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:02 smithi121 ceph-mon[36349]: pgmap v168: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 3.16k op/s 2024-03-26T06:43:04.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:04 smithi038 ceph-mon[32265]: pgmap v169: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 13 KiB/s rd, 29 MiB/s wr, 4.89k op/s 2024-03-26T06:43:04.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:04 smithi121 ceph-mon[36349]: pgmap v169: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 13 KiB/s rd, 29 MiB/s wr, 4.89k op/s 2024-03-26T06:43:04.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:04 smithi170 ceph-mon[36912]: pgmap v169: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 13 KiB/s rd, 29 MiB/s wr, 4.89k op/s 2024-03-26T06:43:06.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:06 smithi038 ceph-mon[32265]: pgmap v170: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 10 KiB/s rd, 19 MiB/s wr, 3.40k op/s 2024-03-26T06:43:06.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:06 smithi121 ceph-mon[36349]: pgmap v170: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 10 KiB/s rd, 19 MiB/s wr, 3.40k op/s 2024-03-26T06:43:06.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:06 smithi170 ceph-mon[36912]: pgmap v170: 97 pgs: 97 active+clean; 21 GiB data, 80 GiB used, 993 GiB / 1.0 TiB avail; 10 KiB/s rd, 19 MiB/s wr, 3.40k op/s 2024-03-26T06:43:08.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:08 smithi038 ceph-mon[32265]: pgmap v171: 97 pgs: 97 active+clean; 22 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 15 KiB/s rd, 37 MiB/s wr, 6.72k op/s 2024-03-26T06:43:08.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:08 smithi121 ceph-mon[36349]: pgmap v171: 97 pgs: 97 active+clean; 22 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 15 KiB/s rd, 37 MiB/s wr, 6.72k op/s 2024-03-26T06:43:08.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:08 smithi170 ceph-mon[36912]: pgmap v171: 97 pgs: 97 active+clean; 22 GiB data, 81 GiB used, 992 GiB / 1.0 TiB avail; 15 KiB/s rd, 37 MiB/s wr, 6.72k op/s 2024-03-26T06:43:10.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:10 smithi038 ceph-mon[32265]: pgmap v172: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 15 KiB/s rd, 63 MiB/s wr, 11.60k op/s 2024-03-26T06:43:10.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:10 smithi121 ceph-mon[36349]: pgmap v172: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 15 KiB/s rd, 63 MiB/s wr, 11.60k op/s 2024-03-26T06:43:10.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:10 smithi170 ceph-mon[36912]: pgmap v172: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 15 KiB/s rd, 63 MiB/s wr, 11.60k op/s 2024-03-26T06:43:12.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:12 smithi038 ceph-mon[32265]: pgmap v173: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 19 KiB/s rd, 65 MiB/s wr, 11.94k op/s 2024-03-26T06:43:12.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:12 smithi121 ceph-mon[36349]: pgmap v173: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 19 KiB/s rd, 65 MiB/s wr, 11.94k op/s 2024-03-26T06:43:12.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:12 smithi170 ceph-mon[36912]: pgmap v173: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 19 KiB/s rd, 65 MiB/s wr, 11.94k op/s 2024-03-26T06:43:14.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:14 smithi038 ceph-mon[32265]: pgmap v174: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 53 KiB/s rd, 86 MiB/s wr, 15.75k op/s 2024-03-26T06:43:14.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:14 smithi121 ceph-mon[36349]: pgmap v174: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 53 KiB/s rd, 86 MiB/s wr, 15.75k op/s 2024-03-26T06:43:14.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:14 smithi170 ceph-mon[36912]: pgmap v174: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 53 KiB/s rd, 86 MiB/s wr, 15.75k op/s 2024-03-26T06:43:16.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:16 smithi038 ceph-mon[32265]: pgmap v175: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 51 KiB/s rd, 72 MiB/s wr, 13.24k op/s 2024-03-26T06:43:16.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:16 smithi121 ceph-mon[36349]: pgmap v175: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 51 KiB/s rd, 72 MiB/s wr, 13.24k op/s 2024-03-26T06:43:16.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:16 smithi170 ceph-mon[36912]: pgmap v175: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 51 KiB/s rd, 72 MiB/s wr, 13.24k op/s 2024-03-26T06:43:18.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:18 smithi038 ceph-mon[32265]: pgmap v176: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 52 KiB/s rd, 72 MiB/s wr, 13.27k op/s 2024-03-26T06:43:18.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:18 smithi121 ceph-mon[36349]: pgmap v176: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 52 KiB/s rd, 72 MiB/s wr, 13.27k op/s 2024-03-26T06:43:18.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:18 smithi170 ceph-mon[36912]: pgmap v176: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 52 KiB/s rd, 72 MiB/s wr, 13.27k op/s 2024-03-26T06:43:20.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:20 smithi038 ceph-mon[32265]: pgmap v177: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 97 KiB/s rd, 55 MiB/s wr, 9.96k op/s 2024-03-26T06:43:20.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:20 smithi121 ceph-mon[36349]: pgmap v177: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 97 KiB/s rd, 55 MiB/s wr, 9.96k op/s 2024-03-26T06:43:20.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:20 smithi170 ceph-mon[36912]: pgmap v177: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 97 KiB/s rd, 55 MiB/s wr, 9.96k op/s 2024-03-26T06:43:22.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:22 smithi038 ceph-mon[32265]: pgmap v178: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 90 KiB/s rd, 27 MiB/s wr, 4.95k op/s 2024-03-26T06:43:22.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:22 smithi121 ceph-mon[36349]: pgmap v178: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 90 KiB/s rd, 27 MiB/s wr, 4.95k op/s 2024-03-26T06:43:22.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:22 smithi170 ceph-mon[36912]: pgmap v178: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 90 KiB/s rd, 27 MiB/s wr, 4.95k op/s 2024-03-26T06:43:24.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:24 smithi038 ceph-mon[32265]: pgmap v179: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 91 KiB/s rd, 25 MiB/s wr, 4.62k op/s 2024-03-26T06:43:24.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:24 smithi121 ceph-mon[36349]: pgmap v179: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 91 KiB/s rd, 25 MiB/s wr, 4.62k op/s 2024-03-26T06:43:24.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:24 smithi170 ceph-mon[36912]: pgmap v179: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 91 KiB/s rd, 25 MiB/s wr, 4.62k op/s 2024-03-26T06:43:26.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:26 smithi038 ceph-mon[32265]: pgmap v180: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 56 KiB/s rd, 145 KiB/s wr, 32 op/s 2024-03-26T06:43:26.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:26 smithi121 ceph-mon[36349]: pgmap v180: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 56 KiB/s rd, 145 KiB/s wr, 32 op/s 2024-03-26T06:43:26.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:26 smithi170 ceph-mon[36912]: pgmap v180: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 56 KiB/s rd, 145 KiB/s wr, 32 op/s 2024-03-26T06:43:28.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:28 smithi038 ceph-mon[32265]: pgmap v181: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 57 KiB/s rd, 145 KiB/s wr, 33 op/s 2024-03-26T06:43:28.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:28 smithi121 ceph-mon[36349]: pgmap v181: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 57 KiB/s rd, 145 KiB/s wr, 33 op/s 2024-03-26T06:43:28.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:28 smithi170 ceph-mon[36912]: pgmap v181: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 57 KiB/s rd, 145 KiB/s wr, 33 op/s 2024-03-26T06:43:30.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:30 smithi038 ceph-mon[32265]: pgmap v182: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 57 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2024-03-26T06:43:30.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:30 smithi121 ceph-mon[36349]: pgmap v182: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 57 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2024-03-26T06:43:30.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:30 smithi170 ceph-mon[36912]: pgmap v182: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 57 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2024-03-26T06:43:32.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:32 smithi038 ceph-mon[32265]: pgmap v183: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:43:32.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:32 smithi121 ceph-mon[36349]: pgmap v183: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:43:32.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:32 smithi170 ceph-mon[36912]: pgmap v183: 97 pgs: 97 active+clean; 22 GiB data, 84 GiB used, 989 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:43:34.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:34 smithi038 ceph-mon[32265]: pgmap v184: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-26T06:43:34.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:34 smithi121 ceph-mon[36349]: pgmap v184: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-26T06:43:34.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:34 smithi170 ceph-mon[36912]: pgmap v184: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 8.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-26T06:43:36.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:36 smithi038 ceph-mon[32265]: pgmap v185: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:43:36.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:36 smithi121 ceph-mon[36349]: pgmap v185: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:43:36.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:36 smithi170 ceph-mon[36912]: pgmap v185: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:43:38.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:38 smithi038 ceph-mon[32265]: pgmap v186: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-26T06:43:38.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:38 smithi121 ceph-mon[36349]: pgmap v186: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-26T06:43:38.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:38 smithi170 ceph-mon[36912]: pgmap v186: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-26T06:43:40.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:40 smithi038 ceph-mon[32265]: pgmap v187: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-26T06:43:40.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:40 smithi121 ceph-mon[36349]: pgmap v187: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-26T06:43:40.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:40 smithi170 ceph-mon[36912]: pgmap v187: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-26T06:43:42.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:42 smithi038 ceph-mon[32265]: pgmap v188: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:42.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:42 smithi121 ceph-mon[36349]: pgmap v188: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:42.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:42 smithi170 ceph-mon[36912]: pgmap v188: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:44.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:44 smithi038 ceph-mon[32265]: pgmap v189: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:43:44.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:44 smithi121 ceph-mon[36349]: pgmap v189: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:43:44.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:44 smithi170 ceph-mon[36912]: pgmap v189: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:43:45.833 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:45 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:43:45.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:45 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:43:45.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:45 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:43:46.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:46 smithi038 ceph-mon[32265]: pgmap v190: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:46.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:46 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:43:46.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:46 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:43:46.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:46 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:43:46.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:46 smithi121 ceph-mon[36349]: pgmap v190: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:46.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:46 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:43:46.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:46 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:43:46.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:46 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:43:46.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:46 smithi170 ceph-mon[36912]: pgmap v190: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:46.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:46 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:43:46.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:46 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:43:46.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:46 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:43:48.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:48 smithi038 ceph-mon[32265]: pgmap v191: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:43:48.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:48 smithi121 ceph-mon[36349]: pgmap v191: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:43:48.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:48 smithi170 ceph-mon[36912]: pgmap v191: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:43:50.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:50 smithi038 ceph-mon[32265]: pgmap v192: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:43:50.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:50 smithi121 ceph-mon[36349]: pgmap v192: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:43:50.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:50 smithi170 ceph-mon[36912]: pgmap v192: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:43:52.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:52 smithi038 ceph-mon[32265]: pgmap v193: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:52.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:52 smithi121 ceph-mon[36349]: pgmap v193: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:52.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:52 smithi170 ceph-mon[36912]: pgmap v193: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:54.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:54 smithi121 ceph-mon[36349]: pgmap v194: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:43:54.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:54 smithi170 ceph-mon[36912]: pgmap v194: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:43:55.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:54 smithi038 ceph-mon[32265]: pgmap v194: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:43:56.886 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:56 smithi038 ceph-mon[32265]: pgmap v195: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:56.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:56 smithi121 ceph-mon[36349]: pgmap v195: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:56.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:56 smithi170 ceph-mon[36912]: pgmap v195: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:43:58.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:43:58 smithi121 ceph-mon[36349]: pgmap v196: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:43:58.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:43:58 smithi170 ceph-mon[36912]: pgmap v196: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:43:59.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:43:58 smithi038 ceph-mon[32265]: pgmap v196: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:44:00.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:00 smithi121 ceph-mon[36349]: pgmap v197: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:44:00.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:00 smithi170 ceph-mon[36912]: pgmap v197: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:44:01.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:00 smithi038 ceph-mon[32265]: pgmap v197: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:44:02.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:02 smithi121 ceph-mon[36349]: pgmap v198: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 19 op/s 2024-03-26T06:44:02.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:02 smithi170 ceph-mon[36912]: pgmap v198: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 19 op/s 2024-03-26T06:44:03.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:02 smithi038 ceph-mon[32265]: pgmap v198: 97 pgs: 97 active+clean; 22 GiB data, 83 GiB used, 990 GiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 19 op/s 2024-03-26T06:44:04.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:04 smithi121 ceph-mon[36349]: pgmap v199: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-26T06:44:04.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:04 smithi170 ceph-mon[36912]: pgmap v199: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-26T06:44:05.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:04 smithi038 ceph-mon[32265]: pgmap v199: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 15 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-26T06:44:06.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:06 smithi121 ceph-mon[36349]: pgmap v200: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-26T06:44:06.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:06 smithi170 ceph-mon[36912]: pgmap v200: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-26T06:44:07.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:06 smithi038 ceph-mon[32265]: pgmap v200: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-26T06:44:08.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:08 smithi121 ceph-mon[36349]: pgmap v201: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-03-26T06:44:08.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:08 smithi121 ceph-mon[36349]: osdmap e91: 12 total, 12 up, 12 in 2024-03-26T06:44:08.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:08 smithi170 ceph-mon[36912]: pgmap v201: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-03-26T06:44:08.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:08 smithi170 ceph-mon[36912]: osdmap e91: 12 total, 12 up, 12 in 2024-03-26T06:44:09.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:08 smithi038 ceph-mon[32265]: pgmap v201: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-03-26T06:44:09.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:08 smithi038 ceph-mon[32265]: osdmap e91: 12 total, 12 up, 12 in 2024-03-26T06:44:10.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:10 smithi121 ceph-mon[36349]: pgmap v203: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-26T06:44:10.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:10 smithi170 ceph-mon[36912]: pgmap v203: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-26T06:44:11.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:10 smithi038 ceph-mon[32265]: pgmap v203: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-26T06:44:12.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:12 smithi121 ceph-mon[36349]: pgmap v204: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-26T06:44:12.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:12 smithi170 ceph-mon[36912]: pgmap v204: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-26T06:44:13.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:12 smithi038 ceph-mon[32265]: pgmap v204: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-26T06:44:14.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:14 smithi121 ceph-mon[36349]: pgmap v205: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:44:14.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:14 smithi170 ceph-mon[36912]: pgmap v205: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:44:15.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:14 smithi038 ceph-mon[32265]: pgmap v205: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:44:16.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:16 smithi170 ceph-mon[36912]: pgmap v206: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:44:17.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:16 smithi038 ceph-mon[32265]: pgmap v206: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:44:17.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:16 smithi121 ceph-mon[36349]: pgmap v206: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:44:18.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:18 smithi170 ceph-mon[36912]: pgmap v207: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:44:18.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:18 smithi170 ceph-mon[36912]: osdmap e92: 12 total, 12 up, 12 in 2024-03-26T06:44:19.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:18 smithi038 ceph-mon[32265]: pgmap v207: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:44:19.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:18 smithi038 ceph-mon[32265]: osdmap e92: 12 total, 12 up, 12 in 2024-03-26T06:44:19.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:18 smithi121 ceph-mon[36349]: pgmap v207: 97 pgs: 97 active+clean; 22 GiB data, 82 GiB used, 991 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:44:19.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:18 smithi121 ceph-mon[36349]: osdmap e92: 12 total, 12 up, 12 in 2024-03-26T06:44:20.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:19 smithi038 ceph-mon[32265]: pgmap v209: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:44:20.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:19 smithi121 ceph-mon[36349]: pgmap v209: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:44:20.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:19 smithi170 ceph-mon[36912]: pgmap v209: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:44:22.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:21 smithi038 ceph-mon[32265]: pgmap v210: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:44:22.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:21 smithi121 ceph-mon[36349]: pgmap v210: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:44:22.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:21 smithi170 ceph-mon[36912]: pgmap v210: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:44:24.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:23 smithi038 ceph-mon[32265]: pgmap v211: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:44:24.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:23 smithi121 ceph-mon[36349]: pgmap v211: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:44:24.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:23 smithi170 ceph-mon[36912]: pgmap v211: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:44:26.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:25 smithi038 ceph-mon[32265]: pgmap v212: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:44:26.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:25 smithi121 ceph-mon[36349]: pgmap v212: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:44:26.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:25 smithi170 ceph-mon[36912]: pgmap v212: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:44:28.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:27 smithi121 ceph-mon[36349]: pgmap v213: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:44:28.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:27 smithi170 ceph-mon[36912]: pgmap v213: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:44:28.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:27 smithi038 ceph-mon[32265]: pgmap v213: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:44:30.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:29 smithi121 ceph-mon[36349]: pgmap v214: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-26T06:44:30.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:29 smithi170 ceph-mon[36912]: pgmap v214: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-26T06:44:30.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:29 smithi038 ceph-mon[32265]: pgmap v214: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-26T06:44:32.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:31 smithi121 ceph-mon[36349]: pgmap v215: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:32.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:31 smithi170 ceph-mon[36912]: pgmap v215: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:32.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:31 smithi038 ceph-mon[32265]: pgmap v215: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:34.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:33 smithi121 ceph-mon[36349]: pgmap v216: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:44:34.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:33 smithi170 ceph-mon[36912]: pgmap v216: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:44:34.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:33 smithi038 ceph-mon[32265]: pgmap v216: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:44:36.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:35 smithi121 ceph-mon[36349]: pgmap v217: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:36.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:35 smithi170 ceph-mon[36912]: pgmap v217: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:36.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:35 smithi038 ceph-mon[32265]: pgmap v217: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:38.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:37 smithi121 ceph-mon[36349]: pgmap v218: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:44:38.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:37 smithi170 ceph-mon[36912]: pgmap v218: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:44:38.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:37 smithi038 ceph-mon[32265]: pgmap v218: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:44:40.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:39 smithi121 ceph-mon[36349]: pgmap v219: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:44:40.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:39 smithi170 ceph-mon[36912]: pgmap v219: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:44:40.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:39 smithi038 ceph-mon[32265]: pgmap v219: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:44:42.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:41 smithi121 ceph-mon[36349]: pgmap v220: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:42.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:41 smithi170 ceph-mon[36912]: pgmap v220: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:42.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:41 smithi038 ceph-mon[32265]: pgmap v220: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:44.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:43 smithi121 ceph-mon[36349]: pgmap v221: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:44:44.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:43 smithi170 ceph-mon[36912]: pgmap v221: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:44:44.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:43 smithi038 ceph-mon[32265]: pgmap v221: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:44:46.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:45 smithi121 ceph-mon[36349]: pgmap v222: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:46.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:45 smithi170 ceph-mon[36912]: pgmap v222: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:46.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:45 smithi038 ceph-mon[32265]: pgmap v222: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:47.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:46 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:44:47.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:46 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:44:47.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:46 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:44:47.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:46 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:44:47.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:46 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:44:47.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:46 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:44:47.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:46 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:44:47.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:46 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:44:47.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:46 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:44:47.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:46 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:44:47.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:46 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:44:47.335 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:46 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:44:48.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:48 smithi121 ceph-mon[36349]: pgmap v223: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:44:48.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:48 smithi170 ceph-mon[36912]: pgmap v223: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:44:49.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:48 smithi038 ceph-mon[32265]: pgmap v223: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:44:50.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:50 smithi121 ceph-mon[36349]: pgmap v224: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:44:50.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:50 smithi170 ceph-mon[36912]: pgmap v224: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:44:51.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:50 smithi038 ceph-mon[32265]: pgmap v224: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:44:52.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:52 smithi121 ceph-mon[36349]: pgmap v225: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:52.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:52 smithi170 ceph-mon[36912]: pgmap v225: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:53.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:52 smithi038 ceph-mon[32265]: pgmap v225: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:54.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:54 smithi121 ceph-mon[36349]: pgmap v226: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:44:54.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:54 smithi170 ceph-mon[36912]: pgmap v226: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:44:55.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:54 smithi038 ceph-mon[32265]: pgmap v226: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:44:56.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:56 smithi170 ceph-mon[36912]: pgmap v227: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:57.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:56 smithi038 ceph-mon[32265]: pgmap v227: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:57.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:56 smithi121 ceph-mon[36349]: pgmap v227: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:44:58.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:44:58 smithi170 ceph-mon[36912]: pgmap v228: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:44:59.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:44:58 smithi038 ceph-mon[32265]: pgmap v228: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:44:59.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:44:58 smithi121 ceph-mon[36349]: pgmap v228: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:45:00.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:00 smithi170 ceph-mon[36912]: pgmap v229: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:45:01.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:00 smithi038 ceph-mon[32265]: pgmap v229: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:45:01.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:00 smithi121 ceph-mon[36349]: pgmap v229: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:45:02.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:02 smithi170 ceph-mon[36912]: pgmap v230: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 19 op/s 2024-03-26T06:45:03.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:02 smithi038 ceph-mon[32265]: pgmap v230: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 19 op/s 2024-03-26T06:45:03.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:02 smithi121 ceph-mon[36349]: pgmap v230: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 19 op/s 2024-03-26T06:45:04.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:04 smithi170 ceph-mon[36912]: pgmap v231: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-26T06:45:05.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:04 smithi038 ceph-mon[32265]: pgmap v231: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-26T06:45:05.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:04 smithi121 ceph-mon[36349]: pgmap v231: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-26T06:45:06.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:05 smithi038 ceph-mon[32265]: pgmap v232: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-26T06:45:06.145 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:05 smithi121 ceph-mon[36349]: pgmap v232: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-26T06:45:06.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:05 smithi170 ceph-mon[36912]: pgmap v232: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-26T06:45:08.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:07 smithi121 ceph-mon[36349]: pgmap v233: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-03-26T06:45:08.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:07 smithi170 ceph-mon[36912]: pgmap v233: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-03-26T06:45:08.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:07 smithi038 ceph-mon[32265]: pgmap v233: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-03-26T06:45:09.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:08 smithi121 ceph-mon[36349]: osdmap e93: 12 total, 12 up, 12 in 2024-03-26T06:45:09.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:08 smithi170 ceph-mon[36912]: osdmap e93: 12 total, 12 up, 12 in 2024-03-26T06:45:09.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:08 smithi038 ceph-mon[32265]: osdmap e93: 12 total, 12 up, 12 in 2024-03-26T06:45:10.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:09 smithi121 ceph-mon[36349]: pgmap v235: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-26T06:45:10.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:09 smithi170 ceph-mon[36912]: pgmap v235: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-26T06:45:10.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:09 smithi038 ceph-mon[32265]: pgmap v235: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-26T06:45:12.146 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:11 smithi121 ceph-mon[36349]: pgmap v236: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-26T06:45:12.168 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:11 smithi170 ceph-mon[36912]: pgmap v236: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-26T06:45:12.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:11 smithi038 ceph-mon[32265]: pgmap v236: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 5.1 KiB/s rd, 28 KiB/s wr, 9 op/s 2024-03-26T06:45:14.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:14 smithi038 ceph-mon[32265]: pgmap v237: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:45:14.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:14 smithi038 ceph-mon[32265]: osdmap e94: 12 total, 12 up, 12 in 2024-03-26T06:45:14.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:14 smithi121 ceph-mon[36349]: pgmap v237: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:45:14.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:14 smithi121 ceph-mon[36349]: osdmap e94: 12 total, 12 up, 12 in 2024-03-26T06:45:14.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:14 smithi170 ceph-mon[36912]: pgmap v237: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:45:14.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:14 smithi170 ceph-mon[36912]: osdmap e94: 12 total, 12 up, 12 in 2024-03-26T06:45:16.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:16 smithi038 ceph-mon[32265]: pgmap v239: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-26T06:45:16.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:16 smithi121 ceph-mon[36349]: pgmap v239: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-26T06:45:16.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:16 smithi170 ceph-mon[36912]: pgmap v239: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-26T06:45:18.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:18 smithi038 ceph-mon[32265]: pgmap v240: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 114 B/s wr, 5 op/s 2024-03-26T06:45:18.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:18 smithi121 ceph-mon[36349]: pgmap v240: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 114 B/s wr, 5 op/s 2024-03-26T06:45:18.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:18 smithi170 ceph-mon[36912]: pgmap v240: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 114 B/s wr, 5 op/s 2024-03-26T06:45:20.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:20 smithi038 ceph-mon[32265]: pgmap v241: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:45:20.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:20 smithi121 ceph-mon[36349]: pgmap v241: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:45:20.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:20 smithi170 ceph-mon[36912]: pgmap v241: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:45:22.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:22 smithi038 ceph-mon[32265]: pgmap v242: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:45:22.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:22 smithi121 ceph-mon[36349]: pgmap v242: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:45:22.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:22 smithi170 ceph-mon[36912]: pgmap v242: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:45:24.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:24 smithi038 ceph-mon[32265]: pgmap v243: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:45:24.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:24 smithi121 ceph-mon[36349]: pgmap v243: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:45:24.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:24 smithi170 ceph-mon[36912]: pgmap v243: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:45:26.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:26 smithi121 ceph-mon[36349]: pgmap v244: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:45:26.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:26 smithi170 ceph-mon[36912]: pgmap v244: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:45:26.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:26 smithi038 ceph-mon[32265]: pgmap v244: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:45:28.386 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:28 smithi038 ceph-mon[32265]: pgmap v245: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-26T06:45:28.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:28 smithi121 ceph-mon[36349]: pgmap v245: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-26T06:45:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:28 smithi170 ceph-mon[36912]: pgmap v245: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-26T06:45:30.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:30 smithi121 ceph-mon[36349]: pgmap v246: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-26T06:45:30.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:30 smithi170 ceph-mon[36912]: pgmap v246: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-26T06:45:30.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:30 smithi038 ceph-mon[32265]: pgmap v246: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-26T06:45:32.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:32 smithi121 ceph-mon[36349]: pgmap v247: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:32 smithi170 ceph-mon[36912]: pgmap v247: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:32.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:32 smithi038 ceph-mon[32265]: pgmap v247: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:34.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:34 smithi121 ceph-mon[36349]: pgmap v248: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:45:34.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:34 smithi170 ceph-mon[36912]: pgmap v248: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:45:34.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:34 smithi038 ceph-mon[32265]: pgmap v248: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:45:36.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:36 smithi121 ceph-mon[36349]: pgmap v249: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:36.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:36 smithi170 ceph-mon[36912]: pgmap v249: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:36.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:36 smithi038 ceph-mon[32265]: pgmap v249: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:38.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:38 smithi121 ceph-mon[36349]: pgmap v250: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:45:38.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:38 smithi170 ceph-mon[36912]: pgmap v250: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:45:38.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:38 smithi038 ceph-mon[32265]: pgmap v250: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:45:40.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:40 smithi121 ceph-mon[36349]: pgmap v251: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:45:40.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:40 smithi170 ceph-mon[36912]: pgmap v251: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:45:40.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:40 smithi038 ceph-mon[32265]: pgmap v251: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:45:42.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:42 smithi121 ceph-mon[36349]: pgmap v252: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:42.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:42 smithi170 ceph-mon[36912]: pgmap v252: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:42.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:42 smithi038 ceph-mon[32265]: pgmap v252: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:44.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:44 smithi121 ceph-mon[36349]: pgmap v253: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:45:44.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:44 smithi170 ceph-mon[36912]: pgmap v253: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:45:44.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:44 smithi038 ceph-mon[32265]: pgmap v253: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:45:46.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:46 smithi121 ceph-mon[36349]: pgmap v254: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-26T06:45:46.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:46 smithi170 ceph-mon[36912]: pgmap v254: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-26T06:45:46.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:46 smithi038 ceph-mon[32265]: pgmap v254: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-26T06:45:47.395 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:47 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:45:47.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:47 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:45:47.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:47 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:45:48.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:48 smithi121 ceph-mon[36349]: pgmap v255: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-26T06:45:48.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:48 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:45:48.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:48 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:45:48.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:48 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:45:48.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:48 smithi170 ceph-mon[36912]: pgmap v255: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-26T06:45:48.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:48 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:45:48.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:48 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:45:48.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:48 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:45:48.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:48 smithi038 ceph-mon[32265]: pgmap v255: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-26T06:45:48.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:48 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:45:48.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:48 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:45:48.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:48 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:45:50.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:50 smithi121 ceph-mon[36349]: pgmap v256: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-26T06:45:50.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:50 smithi170 ceph-mon[36912]: pgmap v256: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-26T06:45:50.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:50 smithi038 ceph-mon[32265]: pgmap v256: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-26T06:45:52.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:52 smithi121 ceph-mon[36349]: pgmap v257: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-26T06:45:52.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:52 smithi170 ceph-mon[36912]: pgmap v257: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-26T06:45:52.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:52 smithi038 ceph-mon[32265]: pgmap v257: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-26T06:45:54.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:54 smithi170 ceph-mon[36912]: pgmap v258: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:45:54.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:54 smithi038 ceph-mon[32265]: pgmap v258: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:45:54.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:54 smithi121 ceph-mon[36349]: pgmap v258: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:45:56.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:56 smithi170 ceph-mon[36912]: pgmap v259: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:56.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:56 smithi038 ceph-mon[32265]: pgmap v259: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:56.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:56 smithi121 ceph-mon[36349]: pgmap v259: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:45:58.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:45:58 smithi170 ceph-mon[36912]: pgmap v260: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:45:58.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:45:58 smithi038 ceph-mon[32265]: pgmap v260: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:45:58.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:45:58 smithi121 ceph-mon[36349]: pgmap v260: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:46:00.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:00 smithi170 ceph-mon[36912]: pgmap v261: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:46:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:00 smithi038 ceph-mon[32265]: pgmap v261: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:46:00.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:00 smithi121 ceph-mon[36349]: pgmap v261: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:46:02.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:02 smithi170 ceph-mon[36912]: pgmap v262: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-03-26T06:46:02.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:02 smithi038 ceph-mon[32265]: pgmap v262: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-03-26T06:46:02.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:02 smithi121 ceph-mon[36349]: pgmap v262: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 11 KiB/s rd, 11 KiB/s wr, 18 op/s 2024-03-26T06:46:04.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:04 smithi038 ceph-mon[32265]: pgmap v263: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 34 KiB/s wr, 24 op/s 2024-03-26T06:46:04.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:04 smithi121 ceph-mon[36349]: pgmap v263: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 34 KiB/s wr, 24 op/s 2024-03-26T06:46:04.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:04 smithi170 ceph-mon[36912]: pgmap v263: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 34 KiB/s wr, 24 op/s 2024-03-26T06:46:06.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:06 smithi038 ceph-mon[32265]: pgmap v264: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 13 KiB/s rd, 34 KiB/s wr, 21 op/s 2024-03-26T06:46:06.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:06 smithi121 ceph-mon[36349]: pgmap v264: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 13 KiB/s rd, 34 KiB/s wr, 21 op/s 2024-03-26T06:46:06.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:06 smithi170 ceph-mon[36912]: pgmap v264: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 13 KiB/s rd, 34 KiB/s wr, 21 op/s 2024-03-26T06:46:08.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:08 smithi038 ceph-mon[32265]: pgmap v265: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-03-26T06:46:08.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:08 smithi038 ceph-mon[32265]: osdmap e95: 12 total, 12 up, 12 in 2024-03-26T06:46:08.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:08 smithi121 ceph-mon[36349]: pgmap v265: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-03-26T06:46:08.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:08 smithi121 ceph-mon[36349]: osdmap e95: 12 total, 12 up, 12 in 2024-03-26T06:46:08.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:08 smithi170 ceph-mon[36912]: pgmap v265: 97 pgs: 97 active+clean; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-03-26T06:46:08.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:08 smithi170 ceph-mon[36912]: osdmap e95: 12 total, 12 up, 12 in 2024-03-26T06:46:10.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:10 smithi038 ceph-mon[32265]: pgmap v267: 97 pgs: 2 active+clean+snaptrim, 89 active+clean, 6 active+clean+snaptrim_wait; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-26T06:46:10.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:10 smithi121 ceph-mon[36349]: pgmap v267: 97 pgs: 2 active+clean+snaptrim, 89 active+clean, 6 active+clean+snaptrim_wait; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-26T06:46:10.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:10 smithi170 ceph-mon[36912]: pgmap v267: 97 pgs: 2 active+clean+snaptrim, 89 active+clean, 6 active+clean+snaptrim_wait; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-26T06:46:12.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:12 smithi038 ceph-mon[32265]: pgmap v268: 97 pgs: 2 active+clean+snaptrim, 89 active+clean, 6 active+clean+snaptrim_wait; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 41 KiB/s wr, 10 op/s 2024-03-26T06:46:12.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:12 smithi121 ceph-mon[36349]: pgmap v268: 97 pgs: 2 active+clean+snaptrim, 89 active+clean, 6 active+clean+snaptrim_wait; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 41 KiB/s wr, 10 op/s 2024-03-26T06:46:12.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:12 smithi170 ceph-mon[36912]: pgmap v268: 97 pgs: 2 active+clean+snaptrim, 89 active+clean, 6 active+clean+snaptrim_wait; 22 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 41 KiB/s wr, 10 op/s 2024-03-26T06:46:14.470 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:14 smithi170 ceph-mon[36912]: pgmap v269: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 17 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-03-26T06:46:14.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:14 smithi038 ceph-mon[32265]: pgmap v269: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 17 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-03-26T06:46:14.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:14 smithi121 ceph-mon[36349]: pgmap v269: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 17 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-03-26T06:46:16.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:16 smithi038 ceph-mon[32265]: pgmap v270: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 17 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-03-26T06:46:16.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:16 smithi121 ceph-mon[36349]: pgmap v270: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 17 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-03-26T06:46:16.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:16 smithi170 ceph-mon[36912]: pgmap v270: 97 pgs: 2 active+clean+snaptrim, 95 active+clean; 17 GiB data, 79 GiB used, 994 GiB / 1.0 TiB avail; 3.4 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-03-26T06:46:18.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:18 smithi038 ceph-mon[32265]: pgmap v271: 97 pgs: 97 active+clean; 17 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:46:18.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:18 smithi038 ceph-mon[32265]: osdmap e96: 12 total, 12 up, 12 in 2024-03-26T06:46:18.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:18 smithi121 ceph-mon[36349]: pgmap v271: 97 pgs: 97 active+clean; 17 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:46:18.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:18 smithi121 ceph-mon[36349]: osdmap e96: 12 total, 12 up, 12 in 2024-03-26T06:46:18.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:18 smithi170 ceph-mon[36912]: pgmap v271: 97 pgs: 97 active+clean; 17 GiB data, 78 GiB used, 995 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:46:18.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:18 smithi170 ceph-mon[36912]: osdmap e96: 12 total, 12 up, 12 in 2024-03-26T06:46:20.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:20 smithi038 ceph-mon[32265]: pgmap v273: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:46:20.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:20 smithi121 ceph-mon[36349]: pgmap v273: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:46:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:20 smithi170 ceph-mon[36912]: pgmap v273: 97 pgs: 97 active+clean; 17 GiB data, 77 GiB used, 996 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:46:22.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:22 smithi038 ceph-mon[32265]: pgmap v274: 97 pgs: 97 active+clean; 17 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:46:22.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:22 smithi121 ceph-mon[36349]: pgmap v274: 97 pgs: 97 active+clean; 17 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:46:22.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:22 smithi170 ceph-mon[36912]: pgmap v274: 97 pgs: 97 active+clean; 17 GiB data, 76 GiB used, 997 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-26T06:46:24.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:24 smithi038 ceph-mon[32265]: pgmap v275: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 999 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:46:24.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:24 smithi121 ceph-mon[36349]: pgmap v275: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 999 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:46:24.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:24 smithi170 ceph-mon[36912]: pgmap v275: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 999 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:46:26.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:26 smithi038 ceph-mon[32265]: pgmap v276: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 999 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:46:26.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:26 smithi121 ceph-mon[36349]: pgmap v276: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 999 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:46:26.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:26 smithi170 ceph-mon[36912]: pgmap v276: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 999 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:46:28.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:28 smithi038 ceph-mon[32265]: pgmap v277: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:46:28.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:28 smithi121 ceph-mon[36349]: pgmap v277: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:46:28.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:28 smithi170 ceph-mon[36912]: pgmap v277: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:46:30.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:30 smithi038 ceph-mon[32265]: pgmap v278: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-26T06:46:30.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:30 smithi121 ceph-mon[36349]: pgmap v278: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-26T06:46:30.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:30 smithi170 ceph-mon[36912]: pgmap v278: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-26T06:46:32.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:32 smithi038 ceph-mon[32265]: pgmap v279: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:32.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:32 smithi121 ceph-mon[36349]: pgmap v279: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:32.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:32 smithi170 ceph-mon[36912]: pgmap v279: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:34.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:34 smithi038 ceph-mon[32265]: pgmap v280: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:46:34.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:34 smithi121 ceph-mon[36349]: pgmap v280: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:46:34.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:34 smithi170 ceph-mon[36912]: pgmap v280: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:46:36.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:36 smithi038 ceph-mon[32265]: pgmap v281: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:36.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:36 smithi121 ceph-mon[36349]: pgmap v281: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:36.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:36 smithi170 ceph-mon[36912]: pgmap v281: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:38.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:38 smithi038 ceph-mon[32265]: pgmap v282: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:46:38.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:38 smithi121 ceph-mon[36349]: pgmap v282: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:46:38.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:38 smithi170 ceph-mon[36912]: pgmap v282: 97 pgs: 97 active+clean; 17 GiB data, 73 GiB used, 1000 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:46:40.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:40 smithi038 ceph-mon[32265]: pgmap v283: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:46:40.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:40 smithi121 ceph-mon[36349]: pgmap v283: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:46:40.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:40 smithi170 ceph-mon[36912]: pgmap v283: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:46:42.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:42 smithi038 ceph-mon[32265]: pgmap v284: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:42.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:42 smithi121 ceph-mon[36349]: pgmap v284: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:42.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:42 smithi170 ceph-mon[36912]: pgmap v284: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:44.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:44 smithi038 ceph-mon[32265]: pgmap v285: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:46:44.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:44 smithi121 ceph-mon[36349]: pgmap v285: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:46:44.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:44 smithi170 ceph-mon[36912]: pgmap v285: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:46:46.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:46 smithi038 ceph-mon[32265]: pgmap v286: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:46.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:46 smithi121 ceph-mon[36349]: pgmap v286: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:46.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:46 smithi170 ceph-mon[36912]: pgmap v286: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:48.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:48 smithi038 ceph-mon[32265]: pgmap v287: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:46:48.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:48 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:46:48.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:48 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:46:48.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:48 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:46:48.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:48 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:46:48.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:48 smithi121 ceph-mon[36349]: pgmap v287: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:46:48.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:48 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:46:48.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:48 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:46:48.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:48 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:46:48.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:48 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:46:48.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:48 smithi170 ceph-mon[36912]: pgmap v287: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:46:48.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:48 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:46:48.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:48 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:46:48.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:48 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:46:48.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:48 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:46:50.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:50 smithi038 ceph-mon[32265]: pgmap v288: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:46:50.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:50 smithi121 ceph-mon[36349]: pgmap v288: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:46:50.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:50 smithi170 ceph-mon[36912]: pgmap v288: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:46:52.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:52 smithi038 ceph-mon[32265]: pgmap v289: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:52.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:52 smithi121 ceph-mon[36349]: pgmap v289: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:52.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:52 smithi170 ceph-mon[36912]: pgmap v289: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:54.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:54 smithi038 ceph-mon[32265]: pgmap v290: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:46:54.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:54 smithi121 ceph-mon[36349]: pgmap v290: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:46:54.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:54 smithi170 ceph-mon[36912]: pgmap v290: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:46:56.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:56 smithi038 ceph-mon[32265]: pgmap v291: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:56.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:56 smithi121 ceph-mon[36349]: pgmap v291: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:56.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:56 smithi170 ceph-mon[36912]: pgmap v291: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:46:58.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:46:58 smithi038 ceph-mon[32265]: pgmap v292: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:46:58.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:46:58 smithi121 ceph-mon[36349]: pgmap v292: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:46:58.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:46:58 smithi170 ceph-mon[36912]: pgmap v292: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:47:00.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:00 smithi038 ceph-mon[32265]: pgmap v293: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:47:00.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:00 smithi121 ceph-mon[36349]: pgmap v293: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:47:00.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:00 smithi170 ceph-mon[36912]: pgmap v293: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:47:02.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:02 smithi038 ceph-mon[32265]: pgmap v294: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-03-26T06:47:02.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:02 smithi121 ceph-mon[36349]: pgmap v294: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-03-26T06:47:02.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:02 smithi170 ceph-mon[36912]: pgmap v294: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 11 KiB/s wr, 7 op/s 2024-03-26T06:47:04.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:04 smithi038 ceph-mon[32265]: pgmap v295: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-03-26T06:47:04.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:04 smithi121 ceph-mon[36349]: pgmap v295: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-03-26T06:47:04.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:04 smithi170 ceph-mon[36912]: pgmap v295: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 34 KiB/s wr, 12 op/s 2024-03-26T06:47:06.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:06 smithi038 ceph-mon[32265]: pgmap v296: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 34 KiB/s wr, 10 op/s 2024-03-26T06:47:06.645 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:06 smithi121 ceph-mon[36349]: pgmap v296: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 34 KiB/s wr, 10 op/s 2024-03-26T06:47:06.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:06 smithi170 ceph-mon[36912]: pgmap v296: 97 pgs: 97 active+clean; 17 GiB data, 72 GiB used, 1001 GiB / 1.0 TiB avail; 5.3 KiB/s rd, 34 KiB/s wr, 10 op/s 2024-03-26T06:47:08.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:08 smithi038 ceph-mon[32265]: pgmap v297: 97 pgs: 97 active+clean; 17 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-03-26T06:47:08.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:08 smithi038 ceph-mon[32265]: osdmap e97: 12 total, 12 up, 12 in 2024-03-26T06:47:08.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:08 smithi121 ceph-mon[36349]: pgmap v297: 97 pgs: 97 active+clean; 17 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-03-26T06:47:08.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:08 smithi121 ceph-mon[36349]: osdmap e97: 12 total, 12 up, 12 in 2024-03-26T06:47:08.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:08 smithi170 ceph-mon[36912]: pgmap v297: 97 pgs: 97 active+clean; 17 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-03-26T06:47:08.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:08 smithi170 ceph-mon[36912]: osdmap e97: 12 total, 12 up, 12 in 2024-03-26T06:47:10.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:10 smithi038 ceph-mon[32265]: pgmap v299: 97 pgs: 97 active+clean; 17 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-26T06:47:10.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:10 smithi121 ceph-mon[36349]: pgmap v299: 97 pgs: 97 active+clean; 17 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-26T06:47:10.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:10 smithi170 ceph-mon[36912]: pgmap v299: 97 pgs: 97 active+clean; 17 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-26T06:47:12.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:12 smithi038 ceph-mon[32265]: pgmap v300: 97 pgs: 97 active+clean; 16 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 14 KiB/s rd, 41 KiB/s wr, 24 op/s 2024-03-26T06:47:12.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:12 smithi121 ceph-mon[36349]: pgmap v300: 97 pgs: 97 active+clean; 16 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 14 KiB/s rd, 41 KiB/s wr, 24 op/s 2024-03-26T06:47:12.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:12 smithi170 ceph-mon[36912]: pgmap v300: 97 pgs: 97 active+clean; 16 GiB data, 71 GiB used, 1002 GiB / 1.0 TiB avail; 14 KiB/s rd, 41 KiB/s wr, 24 op/s 2024-03-26T06:47:14.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:14 smithi038 ceph-mon[32265]: pgmap v301: 97 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 81 active+clean; 13 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 20 op/s 2024-03-26T06:47:14.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:14 smithi121 ceph-mon[36349]: pgmap v301: 97 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 81 active+clean; 13 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 20 op/s 2024-03-26T06:47:14.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:14 smithi170 ceph-mon[36912]: pgmap v301: 97 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 81 active+clean; 13 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 20 op/s 2024-03-26T06:47:16.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:16 smithi038 ceph-mon[32265]: pgmap v302: 97 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 81 active+clean; 13 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 20 op/s 2024-03-26T06:47:16.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:16 smithi121 ceph-mon[36349]: pgmap v302: 97 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 81 active+clean; 13 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 20 op/s 2024-03-26T06:47:16.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:16 smithi170 ceph-mon[36912]: pgmap v302: 97 pgs: 5 active+clean+snaptrim, 11 active+clean+snaptrim_wait, 81 active+clean; 13 GiB data, 69 GiB used, 1004 GiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 20 op/s 2024-03-26T06:47:18.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:18 smithi038 ceph-mon[32265]: pgmap v303: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 13 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:47:18.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:18 smithi121 ceph-mon[36349]: pgmap v303: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 13 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:47:18.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:18 smithi170 ceph-mon[36912]: pgmap v303: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 13 GiB data, 68 GiB used, 1004 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-26T06:47:20.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:20 smithi038 ceph-mon[32265]: pgmap v304: 97 pgs: 97 active+clean; 13 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 88 B/s wr, 5 op/s 2024-03-26T06:47:20.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:20 smithi121 ceph-mon[36349]: pgmap v304: 97 pgs: 97 active+clean; 13 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 88 B/s wr, 5 op/s 2024-03-26T06:47:20.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:20 smithi170 ceph-mon[36912]: pgmap v304: 97 pgs: 97 active+clean; 13 GiB data, 68 GiB used, 1005 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 88 B/s wr, 5 op/s 2024-03-26T06:47:22.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:22 smithi038 ceph-mon[32265]: pgmap v305: 97 pgs: 97 active+clean; 13 GiB data, 67 GiB used, 1005 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:47:22.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:22 smithi121 ceph-mon[36349]: pgmap v305: 97 pgs: 97 active+clean; 13 GiB data, 67 GiB used, 1005 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:47:22.668 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:22 smithi170 ceph-mon[36912]: pgmap v305: 97 pgs: 97 active+clean; 13 GiB data, 67 GiB used, 1005 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-26T06:47:24.340 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:24 smithi038 ceph-mon[32265]: pgmap v306: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-26T06:47:24.341 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:24 smithi038 ceph-mon[32265]: osdmap e98: 12 total, 12 up, 12 in 2024-03-26T06:47:24.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:24 smithi121 ceph-mon[36349]: pgmap v306: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-26T06:47:24.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:24 smithi121 ceph-mon[36349]: osdmap e98: 12 total, 12 up, 12 in 2024-03-26T06:47:24.424 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:24 smithi170 ceph-mon[36912]: pgmap v306: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-26T06:47:24.425 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:24 smithi170 ceph-mon[36912]: osdmap e98: 12 total, 12 up, 12 in 2024-03-26T06:47:26.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:26 smithi038 ceph-mon[32265]: pgmap v308: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:47:26.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:26 smithi121 ceph-mon[36349]: pgmap v308: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:47:26.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:26 smithi170 ceph-mon[36912]: pgmap v308: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:47:28.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:28 smithi121 ceph-mon[36349]: pgmap v309: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:47:28.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:28 smithi170 ceph-mon[36912]: pgmap v309: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:47:28.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:28 smithi038 ceph-mon[32265]: pgmap v309: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-26T06:47:30.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:30 smithi121 ceph-mon[36349]: pgmap v310: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-26T06:47:30.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:30 smithi170 ceph-mon[36912]: pgmap v310: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-26T06:47:30.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:30 smithi038 ceph-mon[32265]: pgmap v310: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-26T06:47:32.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:32 smithi121 ceph-mon[36349]: pgmap v311: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-26T06:47:32.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:32 smithi170 ceph-mon[36912]: pgmap v311: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-26T06:47:32.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:32 smithi038 ceph-mon[32265]: pgmap v311: 97 pgs: 97 active+clean; 13 GiB data, 65 GiB used, 1008 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-26T06:47:34.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:34 smithi121 ceph-mon[36349]: pgmap v312: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-26T06:47:34.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:34 smithi170 ceph-mon[36912]: pgmap v312: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-26T06:47:34.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:34 smithi038 ceph-mon[32265]: pgmap v312: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-26T06:47:34.815 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-26T06:47:34.922 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.916+0000 7f521eb09640 1 -- 172.21.15.38:0/2003843220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52181024d0 msgr2=0x7f52181028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:34.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.916+0000 7f521eb09640 1 --2- 172.21.15.38:0/2003843220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52181024d0 0x7f52181028b0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f520c009900 tx=0x7f520c02bc30 comp rx=0 tx=0).stop 2024-03-26T06:47:34.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 -- 172.21.15.38:0/2003843220 shutdown_connections 2024-03-26T06:47:34.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 --2- 172.21.15.38:0/2003843220 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f521810b940 0x7f521810ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:34.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 --2- 172.21.15.38:0/2003843220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5218102df0 0x7f521810b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:34.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 --2- 172.21.15.38:0/2003843220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52181024d0 0x7f52181028b0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 -- 172.21.15.38:0/2003843220 >> 172.21.15.38:0/2003843220 conn(0x7f5218077320 msgr2=0x7f5218077730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 -- 172.21.15.38:0/2003843220 shutdown_connections 2024-03-26T06:47:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 -- 172.21.15.38:0/2003843220 wait complete. 2024-03-26T06:47:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 Processor -- start 2024-03-26T06:47:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 -- start start 2024-03-26T06:47:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52181024d0 0x7f5218100120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5218102df0 0x7f5218100660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:34.924 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f521810b940 0x7f5218100ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52181116e0 con 0x7f52181024d0 2024-03-26T06:47:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5218111560 con 0x7f521810b940 2024-03-26T06:47:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.917+0000 7f521eb09640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5218111860 con 0x7f5218102df0 2024-03-26T06:47:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521c8a5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52181024d0 0x7f5218100120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521c8a5640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52181024d0 0x7f5218100120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51444/0 (socket says 172.21.15.38:51444) 2024-03-26T06:47:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521c8a5640 1 -- 172.21.15.38:0/1260992951 learned_addr learned my addr 172.21.15.38:0/1260992951 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:47:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521d0a6640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f521810b940 0x7f5218100ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:34.925 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f5217fff640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5218102df0 0x7f5218100660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:34.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521c8a5640 1 -- 172.21.15.38:0/1260992951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5218102df0 msgr2=0x7f5218100660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:34.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521c8a5640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5218102df0 0x7f5218100660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:34.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521c8a5640 1 -- 172.21.15.38:0/1260992951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f521810b940 msgr2=0x7f5218100ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:34.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521c8a5640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f521810b940 0x7f5218100ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:34.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521c8a5640 1 -- 172.21.15.38:0/1260992951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52181a6000 con 0x7f52181024d0 2024-03-26T06:47:34.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521d0a6640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f521810b940 0x7f5218100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:47:34.926 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521c8a5640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52181024d0 0x7f5218100120 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f520c006010 tx=0x7f520c02cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:34.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f5215ffb640 1 -- 172.21.15.38:0/1260992951 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c041070 con 0x7f52181024d0 2024-03-26T06:47:34.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f5215ffb640 1 -- 172.21.15.38:0/1260992951 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f520c004450 con 0x7f52181024d0 2024-03-26T06:47:34.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f521eb09640 1 -- 172.21.15.38:0/1260992951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52181a6290 con 0x7f52181024d0 2024-03-26T06:47:34.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.918+0000 7f5215ffb640 1 -- 172.21.15.38:0/1260992951 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520c003e00 con 0x7f52181024d0 2024-03-26T06:47:34.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.919+0000 7f521eb09640 1 -- 172.21.15.38:0/1260992951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52181a6780 con 0x7f52181024d0 2024-03-26T06:47:34.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.919+0000 7f521eb09640 1 -- 172.21.15.38:0/1260992951 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521806a390 con 0x7f52181024d0 2024-03-26T06:47:34.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.920+0000 7f5215ffb640 1 -- 172.21.15.38:0/1260992951 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f520c044070 con 0x7f52181024d0 2024-03-26T06:47:34.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.921+0000 7f5215ffb640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f52080a2a40 0x7f52080a4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:34.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.921+0000 7f5217fff640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f52080a2a40 0x7f52080a4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:34.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.921+0000 7f5215ffb640 1 -- 172.21.15.38:0/1260992951 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8585+0+0 (secure 0 0 0) 0x7f520c0f08c0 con 0x7f52181024d0 2024-03-26T06:47:34.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.922+0000 7f5217fff640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f52080a2a40 0x7f52080a4f00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5218101b60 tx=0x7f5204005f90 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:34.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:34.926+0000 7f5215ffb640 1 -- 172.21.15.38:0/1260992951 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f520c0bd0a0 con 0x7f52181024d0 2024-03-26T06:47:35.142 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.135+0000 7f521eb09640 1 -- 172.21.15.38:0/1260992951 --> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f52180feaf0 con 0x7f52080a2a40 2024-03-26T06:47:35.146 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.143+0000 7f5215ffb640 1 -- 172.21.15.38:0/1260992951 <== mgr.24883 v2:172.21.15.38:6800/1823987770 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f52180feaf0 con 0x7f52080a2a40 2024-03-26T06:47:35.147 INFO:teuthology.orchestra.run.smithi038.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-26T00:00:00", "created": "2024-03-26T06:37:33", "first": "2024-03-26T06:38:00", "last": "2024-03-26T06:47:00", "last_pruned": "2024-03-26T06:47:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-26T06:47:35.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 -- 172.21.15.38:0/1260992951 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f52080a2a40 msgr2=0x7f52080a4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f52080a2a40 0x7f52080a4f00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5218101b60 tx=0x7f5204005f90 comp rx=0 tx=0).stop 2024-03-26T06:47:35.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 -- 172.21.15.38:0/1260992951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52181024d0 msgr2=0x7f5218100120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52181024d0 0x7f5218100120 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f520c006010 tx=0x7f520c02cb30 comp rx=0 tx=0).stop 2024-03-26T06:47:35.149 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 -- 172.21.15.38:0/1260992951 shutdown_connections 2024-03-26T06:47:35.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f52080a2a40 0x7f52080a4f00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f521810b940 0x7f5218100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5218102df0 0x7f5218100660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 --2- 172.21.15.38:0/1260992951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f52181024d0 0x7f5218100120 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 -- 172.21.15.38:0/1260992951 >> 172.21.15.38:0/1260992951 conn(0x7f5218077320 msgr2=0x7f52180fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:35.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 -- 172.21.15.38:0/1260992951 shutdown_connections 2024-03-26T06:47:35.150 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.145+0000 7f521eb09640 1 -- 172.21.15.38:0/1260992951 wait complete. 2024-03-26T06:47:35.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-26T06:47:35.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.261+0000 7f4322027640 1 -- 172.21.15.38:0/1932275127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f431c100440 msgr2=0x7f431c1008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.261+0000 7f4322027640 1 --2- 172.21.15.38:0/1932275127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f431c100440 0x7f431c1008c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4310009900 tx=0x7f431002bc30 comp rx=0 tx=0).stop 2024-03-26T06:47:35.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.261+0000 7f4322027640 1 -- 172.21.15.38:0/1932275127 shutdown_connections 2024-03-26T06:47:35.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.261+0000 7f4322027640 1 --2- 172.21.15.38:0/1932275127 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431c100e00 0x7f431c10d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.261+0000 7f4322027640 1 --2- 172.21.15.38:0/1932275127 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f431c100440 0x7f431c1008c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.267 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.261+0000 7f4322027640 1 --2- 172.21.15.38:0/1932275127 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f431c105190 0x7f431c105570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.261+0000 7f4322027640 1 -- 172.21.15.38:0/1932275127 >> 172.21.15.38:0/1932275127 conn(0x7f431c077040 msgr2=0x7f431c077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:35.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.261+0000 7f4322027640 1 -- 172.21.15.38:0/1932275127 shutdown_connections 2024-03-26T06:47:35.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.261+0000 7f4322027640 1 -- 172.21.15.38:0/1932275127 wait complete. 2024-03-26T06:47:35.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.262+0000 7f4322027640 1 Processor -- start 2024-03-26T06:47:35.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.262+0000 7f4322027640 1 -- start start 2024-03-26T06:47:35.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.262+0000 7f4322027640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f431c100440 0x7f431c101e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.268 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.262+0000 7f4322027640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f431c100e00 0x7f431c1023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.262+0000 7f4322027640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431c105190 0x7f431c1028e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.262+0000 7f4322027640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c110650 con 0x7f431c100e00 2024-03-26T06:47:35.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.262+0000 7f4322027640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c1104d0 con 0x7f431c100440 2024-03-26T06:47:35.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.262+0000 7f4322027640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431c1107d0 con 0x7f431c105190 2024-03-26T06:47:35.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.262+0000 7f431affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f431c100e00 0x7f431c1023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.262+0000 7f431affd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f431c100e00 0x7f431c1023a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51454/0 (socket says 172.21.15.38:51454) 2024-03-26T06:47:35.269 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.262+0000 7f431affd640 1 -- 172.21.15.38:0/3160234409 learned_addr learned my addr 172.21.15.38:0/3160234409 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:47:35.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f431b7fe640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f431c100440 0x7f431c101e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f431bfff640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431c105190 0x7f431c1028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f431affd640 1 -- 172.21.15.38:0/3160234409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431c105190 msgr2=0x7f431c1028e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f431affd640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431c105190 0x7f431c1028e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f431affd640 1 -- 172.21.15.38:0/3160234409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f431c100440 msgr2=0x7f431c101e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f431affd640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f431c100440 0x7f431c101e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.270 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f431affd640 1 -- 172.21.15.38:0/3160234409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c1a60d0 con 0x7f431c100e00 2024-03-26T06:47:35.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f431affd640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f431c100e00 0x7f431c1023a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4310005940 tx=0x7f43100042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:35.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f4318ff9640 1 -- 172.21.15.38:0/3160234409 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310041070 con 0x7f431c100e00 2024-03-26T06:47:35.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f4318ff9640 1 -- 172.21.15.38:0/3160234409 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f431002ccc0 con 0x7f431c100e00 2024-03-26T06:47:35.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f4322027640 1 -- 172.21.15.38:0/3160234409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431c1a6360 con 0x7f431c100e00 2024-03-26T06:47:35.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f4318ff9640 1 -- 172.21.15.38:0/3160234409 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310003d80 con 0x7f431c100e00 2024-03-26T06:47:35.271 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.263+0000 7f4322027640 1 -- 172.21.15.38:0/3160234409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431c1a6850 con 0x7f431c100e00 2024-03-26T06:47:35.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.264+0000 7f4322027640 1 -- 172.21.15.38:0/3160234409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431c04f050 con 0x7f431c100e00 2024-03-26T06:47:35.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.265+0000 7f4318ff9640 1 -- 172.21.15.38:0/3160234409 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4310044070 con 0x7f431c100e00 2024-03-26T06:47:35.272 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.266+0000 7f4318ff9640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f43040a2a10 0x7f43040a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.266+0000 7f431b7fe640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f43040a2a10 0x7f43040a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.266+0000 7f4318ff9640 1 -- 172.21.15.38:0/3160234409 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8585+0+0 (secure 0 0 0) 0x7f43100f0870 con 0x7f431c100e00 2024-03-26T06:47:35.273 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.267+0000 7f431b7fe640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f43040a2a10 0x7f43040a4ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f430c00bb60 tx=0x7f430c00b390 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:35.274 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.271+0000 7f4318ff9640 1 -- 172.21.15.38:0/3160234409 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f43100bd050 con 0x7f431c100e00 2024-03-26T06:47:35.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.476+0000 7f4322027640 1 -- 172.21.15.38:0/3160234409 --> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f431c061730 con 0x7f43040a2a10 2024-03-26T06:47:35.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.484+0000 7f4318ff9640 1 -- 172.21.15.38:0/3160234409 <== mgr.24883 v2:172.21.15.38:6800/1823987770 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f431c061730 con 0x7f43040a2a10 2024-03-26T06:47:35.488 INFO:teuthology.orchestra.run.smithi038.stdout:/ 1m 6m3h 2024-03-26T06:47:35.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 -- 172.21.15.38:0/3160234409 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f43040a2a10 msgr2=0x7f43040a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f43040a2a10 0x7f43040a4ed0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f430c00bb60 tx=0x7f430c00b390 comp rx=0 tx=0).stop 2024-03-26T06:47:35.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 -- 172.21.15.38:0/3160234409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f431c100e00 msgr2=0x7f431c1023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f431c100e00 0x7f431c1023a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4310005940 tx=0x7f43100042f0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.490 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 -- 172.21.15.38:0/3160234409 shutdown_connections 2024-03-26T06:47:35.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f43040a2a10 0x7f43040a4ed0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f431c105190 0x7f431c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f431c100e00 0x7f431c1023a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 --2- 172.21.15.38:0/3160234409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f431c100440 0x7f431c101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 -- 172.21.15.38:0/3160234409 >> 172.21.15.38:0/3160234409 conn(0x7f431c077040 msgr2=0x7f431c068bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:35.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 -- 172.21.15.38:0/3160234409 shutdown_connections 2024-03-26T06:47:35.491 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.486+0000 7f4322027640 1 -- 172.21.15.38:0/3160234409 wait complete. 2024-03-26T06:47:35.501 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-26T06:47:35.512 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:47:35.619 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.613+0000 7ff6d54c6640 1 -- 172.21.15.38:0/1147244874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d0105530 msgr2=0x7ff6d010d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.613+0000 7ff6d54c6640 1 --2- 172.21.15.38:0/1147244874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d0105530 0x7ff6d010d100 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009900 tx=0x7ff6c802bc30 comp rx=0 tx=0).stop 2024-03-26T06:47:35.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.613+0000 7ff6d54c6640 1 -- 172.21.15.38:0/1147244874 shutdown_connections 2024-03-26T06:47:35.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.613+0000 7ff6d54c6640 1 --2- 172.21.15.38:0/1147244874 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d0105530 0x7ff6d010d100 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.613+0000 7ff6d54c6640 1 --2- 172.21.15.38:0/1147244874 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d0100e10 0x7ff6d0104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.620 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.613+0000 7ff6d54c6640 1 --2- 172.21.15.38:0/1147244874 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6d0100460 0x7ff6d0100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.613+0000 7ff6d54c6640 1 -- 172.21.15.38:0/1147244874 >> 172.21.15.38:0/1147244874 conn(0x7ff6d00772f0 msgr2=0x7ff6d0077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:35.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.613+0000 7ff6d54c6640 1 -- 172.21.15.38:0/1147244874 shutdown_connections 2024-03-26T06:47:35.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.613+0000 7ff6d54c6640 1 -- 172.21.15.38:0/1147244874 wait complete. 2024-03-26T06:47:35.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.613+0000 7ff6d54c6640 1 Processor -- start 2024-03-26T06:47:35.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6d54c6640 1 -- start start 2024-03-26T06:47:35.621 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6d54c6640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d0100460 0x7ff6d0199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6d54c6640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d0105530 0x7ff6d019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6d54c6640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6d019e6a0 0x7ff6d019eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6d54c6640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d01105f0 con 0x7ff6d019e6a0 2024-03-26T06:47:35.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6d54c6640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d0110470 con 0x7ff6d0100460 2024-03-26T06:47:35.622 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6d54c6640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d0110770 con 0x7ff6d0105530 2024-03-26T06:47:35.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6cf7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6d019e6a0 0x7ff6d019eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6ceffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d0100460 0x7ff6d0199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6cf7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6d019e6a0 0x7ff6d019eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51476/0 (socket says 172.21.15.38:51476) 2024-03-26T06:47:35.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6ceffd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d0100460 0x7ff6d0199de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.38:35950/0 (socket says 172.21.15.38:35950) 2024-03-26T06:47:35.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6cf7fe640 1 -- 172.21.15.38:0/3983415520 learned_addr learned my addr 172.21.15.38:0/3983415520 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:47:35.623 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6ce7fc640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d0105530 0x7ff6d019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6cf7fe640 1 -- 172.21.15.38:0/3983415520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d0105530 msgr2=0x7ff6d019a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6cf7fe640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d0105530 0x7ff6d019a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6cf7fe640 1 -- 172.21.15.38:0/3983415520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d0100460 msgr2=0x7ff6d0199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6cf7fe640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d0100460 0x7ff6d0199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6cf7fe640 1 -- 172.21.15.38:0/3983415520 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d019f190 con 0x7ff6d019e6a0 2024-03-26T06:47:35.624 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6ceffd640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d0100460 0x7ff6d0199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:47:35.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6ce7fc640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d0105530 0x7ff6d019a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:47:35.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.614+0000 7ff6cf7fe640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6d019e6a0 0x7ff6d019eb50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009230 tx=0x7ff6c802cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:35.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.615+0000 7ff6b7fff640 1 -- 172.21.15.38:0/3983415520 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8041070 con 0x7ff6d019e6a0 2024-03-26T06:47:35.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.615+0000 7ff6d54c6640 1 -- 172.21.15.38:0/3983415520 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d019f420 con 0x7ff6d019e6a0 2024-03-26T06:47:35.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.615+0000 7ff6b7fff640 1 -- 172.21.15.38:0/3983415520 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6c8004450 con 0x7ff6d019e6a0 2024-03-26T06:47:35.625 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.615+0000 7ff6b7fff640 1 -- 172.21.15.38:0/3983415520 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8003d90 con 0x7ff6d019e6a0 2024-03-26T06:47:35.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.615+0000 7ff6d54c6640 1 -- 172.21.15.38:0/3983415520 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d01a6bf0 con 0x7ff6d019e6a0 2024-03-26T06:47:35.626 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.616+0000 7ff6d54c6640 1 -- 172.21.15.38:0/3983415520 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d00680e0 con 0x7ff6d019e6a0 2024-03-26T06:47:35.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.617+0000 7ff6b7fff640 1 -- 172.21.15.38:0/3983415520 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff6c8044070 con 0x7ff6d019e6a0 2024-03-26T06:47:35.627 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.617+0000 7ff6b7fff640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6b80a2970 0x7ff6b80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.617+0000 7ff6ceffd640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6b80a2970 0x7ff6b80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.617+0000 7ff6b7fff640 1 -- 172.21.15.38:0/3983415520 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8585+0+0 (secure 0 0 0) 0x7ff6c80f0960 con 0x7ff6d019e6a0 2024-03-26T06:47:35.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.618+0000 7ff6ceffd640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6b80a2970 0x7ff6b80a4e30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0005fd0 tx=0x7ff6c0009500 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:35.628 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.622+0000 7ff6b7fff640 1 -- 172.21.15.38:0/3983415520 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff6c80bd790 con 0x7ff6d019e6a0 2024-03-26T06:47:35.822 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.818+0000 7ff6d54c6640 1 -- 172.21.15.38:0/3983415520 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6d0100e10 con 0x7ff6d019e6a0 2024-03-26T06:47:35.823 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.820+0000 7ff6b7fff640 1 -- 172.21.15.38:0/3983415520 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+23164 (secure 0 0 0) 0x7ff6c80bd090 con 0x7ff6d019e6a0 2024-03-26T06:47:35.824 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:47:35.824 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":98,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:47:23.073912+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"97","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":97,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/2716021457":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/2750459261":"2024-03-27T06:37:32.618701+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/1675065388":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/4027757314":"2024-03-27T06:37:32.618701+0000","172.21.15.38:6800/2716021457":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/3320689790":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/3570258347":"2024-03-27T06:37:32.618701+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:47:35.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 -- 172.21.15.38:0/3983415520 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6b80a2970 msgr2=0x7ff6b80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6b80a2970 0x7ff6b80a4e30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6c0005fd0 tx=0x7ff6c0009500 comp rx=0 tx=0).stop 2024-03-26T06:47:35.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 -- 172.21.15.38:0/3983415520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6d019e6a0 msgr2=0x7ff6d019eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6d019e6a0 0x7ff6d019eb50 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009230 tx=0x7ff6c802cba0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 -- 172.21.15.38:0/3983415520 shutdown_connections 2024-03-26T06:47:35.826 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff6b80a2970 0x7ff6b80a4e30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6d019e6a0 0x7ff6d019eb50 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff6d0105530 0x7ff6d019a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 --2- 172.21.15.38:0/3983415520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d0100460 0x7ff6d0199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 -- 172.21.15.38:0/3983415520 >> 172.21.15.38:0/3983415520 conn(0x7ff6d00772f0 msgr2=0x7ff6d00fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:35.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 -- 172.21.15.38:0/3983415520 shutdown_connections 2024-03-26T06:47:35.827 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.822+0000 7ff6d54c6640 1 -- 172.21.15.38:0/3983415520 wait complete. 2024-03-26T06:47:35.837 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T06:30:52.485897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-26T06:35:24.501259+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-26T06:35:25.675299+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '97', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 97, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T06:47:35.838 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T06:47:35.947 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.942+0000 7fed0c081640 1 -- 172.21.15.38:0/1706249392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed041054a0 msgr2=0x7fed0410d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.942+0000 7fed0c081640 1 --2- 172.21.15.38:0/1706249392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed041054a0 0x7fed0410d070 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fecfc009ca0 tx=0x7fecfc02c0f0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.942+0000 7fed0c081640 1 -- 172.21.15.38:0/1706249392 shutdown_connections 2024-03-26T06:47:35.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.942+0000 7fed0c081640 1 --2- 172.21.15.38:0/1706249392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed041054a0 0x7fed0410d070 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.942+0000 7fed0c081640 1 --2- 172.21.15.38:0/1706249392 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed04100d80 0x7fed04104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.942+0000 7fed0c081640 1 --2- 172.21.15.38:0/1706249392 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed041003d0 0x7fed041007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.942+0000 7fed0c081640 1 -- 172.21.15.38:0/1706249392 >> 172.21.15.38:0/1706249392 conn(0x7fed040772f0 msgr2=0x7fed04077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:35.948 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.942+0000 7fed0c081640 1 -- 172.21.15.38:0/1706249392 shutdown_connections 2024-03-26T06:47:35.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.942+0000 7fed0c081640 1 -- 172.21.15.38:0/1706249392 wait complete. 2024-03-26T06:47:35.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.942+0000 7fed0c081640 1 Processor -- start 2024-03-26T06:47:35.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.942+0000 7fed0c081640 1 -- start start 2024-03-26T06:47:35.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0c081640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed041003d0 0x7fed041989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0c081640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed041054a0 0x7fed04198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0c081640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed04199440 0x7fed041a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0c081640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed04110560 con 0x7fed041054a0 2024-03-26T06:47:35.949 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0c081640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed041103e0 con 0x7fed041003d0 2024-03-26T06:47:35.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0c081640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed041106e0 con 0x7fed04199440 2024-03-26T06:47:35.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0961c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed041054a0 0x7fed04198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0961c640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed041054a0 0x7fed04198f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51484/0 (socket says 172.21.15.38:51484) 2024-03-26T06:47:35.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0961c640 1 -- 172.21.15.38:0/66972138 learned_addr learned my addr 172.21.15.38:0/66972138 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:47:35.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed09e1d640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed041003d0 0x7fed041989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0a61e640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed04199440 0x7fed041a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0961c640 1 -- 172.21.15.38:0/66972138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed04199440 msgr2=0x7fed041a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.950 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0961c640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed04199440 0x7fed041a3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0961c640 1 -- 172.21.15.38:0/66972138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed041003d0 msgr2=0x7fed041989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:35.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0961c640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed041003d0 0x7fed041989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:35.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0961c640 1 -- 172.21.15.38:0/66972138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed041a41e0 con 0x7fed041054a0 2024-03-26T06:47:35.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0961c640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed041054a0 0x7fed04198f00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fecf400e9f0 tx=0x7fecf400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:35.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fecfaffd640 1 -- 172.21.15.38:0/66972138 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf400ccc0 con 0x7fed041054a0 2024-03-26T06:47:35.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fecfaffd640 1 -- 172.21.15.38:0/66972138 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecf4005500 con 0x7fed041054a0 2024-03-26T06:47:35.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fed0c081640 1 -- 172.21.15.38:0/66972138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed041a44d0 con 0x7fed041054a0 2024-03-26T06:47:35.951 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.943+0000 7fecfaffd640 1 -- 172.21.15.38:0/66972138 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf4010690 con 0x7fed041054a0 2024-03-26T06:47:35.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.944+0000 7fed0c081640 1 -- 172.21.15.38:0/66972138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed041a49d0 con 0x7fed041054a0 2024-03-26T06:47:35.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.944+0000 7fed0c081640 1 -- 172.21.15.38:0/66972138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed040680e0 con 0x7fed041054a0 2024-03-26T06:47:35.952 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.946+0000 7fecfaffd640 1 -- 172.21.15.38:0/66972138 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fecf40046c0 con 0x7fed041054a0 2024-03-26T06:47:35.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.946+0000 7fecfaffd640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fecf00a29a0 0x7fecf00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:35.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.946+0000 7fed09e1d640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fecf00a29a0 0x7fecf00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:35.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.946+0000 7fecfaffd640 1 -- 172.21.15.38:0/66972138 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8585+0+0 (secure 0 0 0) 0x7fecf40cf330 con 0x7fed041054a0 2024-03-26T06:47:35.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.947+0000 7fed09e1d640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fecf00a29a0 0x7fecf00a4e60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fed00009bd0 tx=0x7fed00005f90 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:35.953 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:35.950+0000 7fecfaffd640 1 -- 172.21.15.38:0/66972138 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fecf4053260 con 0x7fed041054a0 2024-03-26T06:47:36.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.133+0000 7fed0c081640 1 -- 172.21.15.38:0/66972138 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed041086f0 con 0x7fed041054a0 2024-03-26T06:47:36.136 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.133+0000 7fecfaffd640 1 -- 172.21.15.38:0/66972138 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7fecf409c050 con 0x7fed041054a0 2024-03-26T06:47:36.137 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-03-26T06:47:36.138 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 -- 172.21.15.38:0/66972138 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fecf00a29a0 msgr2=0x7fecf00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fecf00a29a0 0x7fecf00a4e60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fed00009bd0 tx=0x7fed00005f90 comp rx=0 tx=0).stop 2024-03-26T06:47:36.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 -- 172.21.15.38:0/66972138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed041054a0 msgr2=0x7fed04198f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed041054a0 0x7fed04198f00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fecf400e9f0 tx=0x7fecf400eeb0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 -- 172.21.15.38:0/66972138 shutdown_connections 2024-03-26T06:47:36.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fecf00a29a0 0x7fecf00a4e60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fed04199440 0x7fed041a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.139 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fed041054a0 0x7fed04198f00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 --2- 172.21.15.38:0/66972138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed041003d0 0x7fed041989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 -- 172.21.15.38:0/66972138 >> 172.21.15.38:0/66972138 conn(0x7fed040772f0 msgr2=0x7fed040fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:36.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 -- 172.21.15.38:0/66972138 shutdown_connections 2024-03-26T06:47:36.140 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.135+0000 7fed0c081640 1 -- 172.21.15.38:0/66972138 wait complete. 2024-03-26T06:47:36.150 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-26T06:47:36.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.254+0000 7f72d44a4640 1 -- 172.21.15.38:0/2315165409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72cc100d80 msgr2=0x7f72cc104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.254+0000 7f72d44a4640 1 --2- 172.21.15.38:0/2315165409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72cc100d80 0x7f72cc104dc0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009900 tx=0x7f72bc02bc30 comp rx=0 tx=0).stop 2024-03-26T06:47:36.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.254+0000 7f72d44a4640 1 -- 172.21.15.38:0/2315165409 shutdown_connections 2024-03-26T06:47:36.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.254+0000 7f72d44a4640 1 --2- 172.21.15.38:0/2315165409 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72cc1054a0 0x7f72cc10d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.254+0000 7f72d44a4640 1 --2- 172.21.15.38:0/2315165409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72cc100d80 0x7f72cc104dc0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.258 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.254+0000 7f72d44a4640 1 --2- 172.21.15.38:0/2315165409 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72cc1003d0 0x7f72cc1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.254+0000 7f72d44a4640 1 -- 172.21.15.38:0/2315165409 >> 172.21.15.38:0/2315165409 conn(0x7f72cc0772f0 msgr2=0x7f72cc077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:36.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.254+0000 7f72d44a4640 1 -- 172.21.15.38:0/2315165409 shutdown_connections 2024-03-26T06:47:36.259 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.254+0000 7f72d44a4640 1 -- 172.21.15.38:0/2315165409 wait complete. 2024-03-26T06:47:36.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d44a4640 1 Processor -- start 2024-03-26T06:47:36.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d44a4640 1 -- start start 2024-03-26T06:47:36.261 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d44a4640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72cc1003d0 0x7f72cc196780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d44a4640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72cc1054a0 0x7f72cc196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d44a4640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72cc197200 0x7f72cc1a5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d44a4640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc110510 con 0x7f72cc1054a0 2024-03-26T06:47:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d44a4640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc110390 con 0x7f72cc1003d0 2024-03-26T06:47:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d44a4640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc110690 con 0x7f72cc197200 2024-03-26T06:47:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d1a3f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72cc1054a0 0x7f72cc196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d1a3f640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72cc1054a0 0x7f72cc196cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51506/0 (socket says 172.21.15.38:51506) 2024-03-26T06:47:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d1a3f640 1 -- 172.21.15.38:0/3249150555 learned_addr learned my addr 172.21.15.38:0/3249150555 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:47:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d2240640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72cc1003d0 0x7f72cc196780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.255+0000 7f72d2a41640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72cc197200 0x7f72cc1a5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.256+0000 7f72d1a3f640 1 -- 172.21.15.38:0/3249150555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72cc197200 msgr2=0x7f72cc1a5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.256+0000 7f72d1a3f640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72cc197200 0x7f72cc1a5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.256+0000 7f72d1a3f640 1 -- 172.21.15.38:0/3249150555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72cc1003d0 msgr2=0x7f72cc196780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.256+0000 7f72d1a3f640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72cc1003d0 0x7f72cc196780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.256+0000 7f72d1a3f640 1 -- 172.21.15.38:0/3249150555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc1a63c0 con 0x7f72cc1054a0 2024-03-26T06:47:36.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.256+0000 7f72d1a3f640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72cc1054a0 0x7f72cc196cc0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f72bc02cbf0 tx=0x7f72bc02cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:36.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.256+0000 7f72c37fe640 1 -- 172.21.15.38:0/3249150555 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc041020 con 0x7f72cc1054a0 2024-03-26T06:47:36.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.256+0000 7f72d44a4640 1 -- 172.21.15.38:0/3249150555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72cc1a6650 con 0x7f72cc1054a0 2024-03-26T06:47:36.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.256+0000 7f72c37fe640 1 -- 172.21.15.38:0/3249150555 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72bc004460 con 0x7f72cc1054a0 2024-03-26T06:47:36.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.256+0000 7f72c37fe640 1 -- 172.21.15.38:0/3249150555 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72bc003e50 con 0x7f72cc1054a0 2024-03-26T06:47:36.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.256+0000 7f72d44a4640 1 -- 172.21.15.38:0/3249150555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72cc1a6b90 con 0x7f72cc1054a0 2024-03-26T06:47:36.264 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.257+0000 7f72d44a4640 1 -- 172.21.15.38:0/3249150555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72cc0680e0 con 0x7f72cc1054a0 2024-03-26T06:47:36.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.258+0000 7f72c37fe640 1 -- 172.21.15.38:0/3249150555 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f72bc044020 con 0x7f72cc1054a0 2024-03-26T06:47:36.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.259+0000 7f72c37fe640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72b80a2b10 0x7f72b80a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.265 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.259+0000 7f72d2240640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72b80a2b10 0x7f72b80a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.259+0000 7f72c37fe640 1 -- 172.21.15.38:0/3249150555 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8585+0+0 (secure 0 0 0) 0x7f72bc0f1110 con 0x7f72cc1054a0 2024-03-26T06:47:36.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.259+0000 7f72d2240640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72b80a2b10 0x7f72b80a4fd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f72c800abe0 tx=0x7f72c800a2d0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:36.266 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.263+0000 7f72c37fe640 1 -- 172.21.15.38:0/3249150555 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f72bc04b020 con 0x7f72cc1054a0 2024-03-26T06:47:36.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:36 smithi121 ceph-mon[36349]: pgmap v313: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:47:36.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:36 smithi121 ceph-mon[36349]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:47:36.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:36 smithi121 ceph-mon[36349]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:47:36.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:36 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3983415520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:47:36.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:36 smithi170 ceph-mon[36912]: pgmap v313: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:47:36.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:36 smithi170 ceph-mon[36912]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:47:36.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:36 smithi170 ceph-mon[36912]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:47:36.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:36 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3983415520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:47:36.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.463+0000 7f72d44a4640 1 -- 172.21.15.38:0/3249150555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f72cc100d80 con 0x7f72cc1054a0 2024-03-26T06:47:36.466 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.463+0000 7f72c37fe640 1 -- 172.21.15.38:0/3249150555 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f72bc0a6e20 con 0x7f72cc1054a0 2024-03-26T06:47:36.467 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-03-26T06:47:36.468 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 -- 172.21.15.38:0/3249150555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72b80a2b10 msgr2=0x7f72b80a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72b80a2b10 0x7f72b80a4fd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f72c800abe0 tx=0x7f72c800a2d0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 -- 172.21.15.38:0/3249150555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72cc1054a0 msgr2=0x7f72cc196cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72cc1054a0 0x7f72cc196cc0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f72bc02cbf0 tx=0x7f72bc02cc20 comp rx=0 tx=0).stop 2024-03-26T06:47:36.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 -- 172.21.15.38:0/3249150555 shutdown_connections 2024-03-26T06:47:36.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f72b80a2b10 0x7f72b80a4fd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f72cc197200 0x7f72cc1a5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.469 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72cc1054a0 0x7f72cc196cc0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 --2- 172.21.15.38:0/3249150555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72cc1003d0 0x7f72cc196780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 -- 172.21.15.38:0/3249150555 >> 172.21.15.38:0/3249150555 conn(0x7f72cc0772f0 msgr2=0x7f72cc0fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:36.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 -- 172.21.15.38:0/3249150555 shutdown_connections 2024-03-26T06:47:36.470 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.465+0000 7f72d44a4640 1 -- 172.21.15.38:0/3249150555 wait complete. 2024-03-26T06:47:36.478 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:36 smithi038 ceph-mon[32265]: pgmap v313: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-26T06:47:36.478 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:36 smithi038 ceph-mon[32265]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:47:36.479 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:36 smithi038 ceph-mon[32265]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T06:47:36.479 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:36 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3983415520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:47:36.480 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-26T06:47:36.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.584+0000 7fe27dcd1640 1 -- 172.21.15.38:0/4292003620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe27810b930 msgr2=0x7fe27810dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.584+0000 7fe27dcd1640 1 --2- 172.21.15.38:0/4292003620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe27810b930 0x7fe27810dda0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe270009ca0 tx=0x7fe27002c180 comp rx=0 tx=0).stop 2024-03-26T06:47:36.588 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.584+0000 7fe27dcd1640 1 -- 172.21.15.38:0/4292003620 shutdown_connections 2024-03-26T06:47:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.584+0000 7fe27dcd1640 1 --2- 172.21.15.38:0/4292003620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe27810b930 0x7fe27810dda0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.584+0000 7fe27dcd1640 1 --2- 172.21.15.38:0/4292003620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe278102e80 0x7fe27810b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.584+0000 7fe27dcd1640 1 --2- 172.21.15.38:0/4292003620 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278102560 0x7fe278102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.584+0000 7fe27dcd1640 1 -- 172.21.15.38:0/4292003620 >> 172.21.15.38:0/4292003620 conn(0x7fe278077350 msgr2=0x7fe278077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.584+0000 7fe27dcd1640 1 -- 172.21.15.38:0/4292003620 shutdown_connections 2024-03-26T06:47:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.584+0000 7fe27dcd1640 1 -- 172.21.15.38:0/4292003620 wait complete. 2024-03-26T06:47:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe27dcd1640 1 Processor -- start 2024-03-26T06:47:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe27dcd1640 1 -- start start 2024-03-26T06:47:36.591 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe27dcd1640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278102560 0x7fe2781001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe27dcd1640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe278102e80 0x7fe2781006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe27dcd1640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe27810b930 0x7fe278100c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe27dcd1640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2781116c0 con 0x7fe27810b930 2024-03-26T06:47:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe27dcd1640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe278111540 con 0x7fe278102e80 2024-03-26T06:47:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe27dcd1640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe278111840 con 0x7fe278102560 2024-03-26T06:47:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe277fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe27810b930 0x7fe278100c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe277fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe27810b930 0x7fe278100c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51524/0 (socket says 172.21.15.38:51524) 2024-03-26T06:47:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe277fff640 1 -- 172.21.15.38:0/134569495 learned_addr learned my addr 172.21.15.38:0/134569495 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:47:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.585+0000 7fe276ffd640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe278102e80 0x7fe2781006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe2777fe640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278102560 0x7fe2781001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe277fff640 1 -- 172.21.15.38:0/134569495 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278102560 msgr2=0x7fe2781001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe277fff640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278102560 0x7fe2781001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe277fff640 1 -- 172.21.15.38:0/134569495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe278102e80 msgr2=0x7fe2781006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe277fff640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe278102e80 0x7fe2781006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe277fff640 1 -- 172.21.15.38:0/134569495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2781a6000 con 0x7fe27810b930 2024-03-26T06:47:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe276ffd640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe278102e80 0x7fe2781006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:47:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe277fff640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe27810b930 0x7fe278100c30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe27002c790 tx=0x7fe270035ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe274ff9640 1 -- 172.21.15.38:0/134569495 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe270039040 con 0x7fe27810b930 2024-03-26T06:47:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe274ff9640 1 -- 172.21.15.38:0/134569495 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe270044070 con 0x7fe27810b930 2024-03-26T06:47:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe274ff9640 1 -- 172.21.15.38:0/134569495 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2700329f0 con 0x7fe27810b930 2024-03-26T06:47:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe27dcd1640 1 -- 172.21.15.38:0/134569495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2781a6290 con 0x7fe27810b930 2024-03-26T06:47:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.586+0000 7fe27dcd1640 1 -- 172.21.15.38:0/134569495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2781a6730 con 0x7fe27810b930 2024-03-26T06:47:36.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.587+0000 7fe27dcd1640 1 -- 172.21.15.38:0/134569495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe278068530 con 0x7fe27810b930 2024-03-26T06:47:36.595 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.588+0000 7fe274ff9640 1 -- 172.21.15.38:0/134569495 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fe270032b90 con 0x7fe27810b930 2024-03-26T06:47:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.589+0000 7fe274ff9640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2600a29a0 0x7fe2600a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.589+0000 7fe2777fe640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2600a29a0 0x7fe2600a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.589+0000 7fe274ff9640 1 -- 172.21.15.38:0/134569495 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8585+0+0 (secure 0 0 0) 0x7fe2700f1960 con 0x7fe27810b930 2024-03-26T06:47:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.589+0000 7fe2777fe640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2600a29a0 0x7fe2600a4e60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe268005e10 tx=0x7fe268005da0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.593+0000 7fe274ff9640 1 -- 172.21.15.38:0/134569495 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe2700be790 con 0x7fe27810b930 2024-03-26T06:47:36.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.790+0000 7fe27dcd1640 1 -- 172.21.15.38:0/134569495 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe2781a6a50 con 0x7fe27810b930 2024-03-26T06:47:36.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.790+0000 7fe274ff9640 1 -- 172.21.15.38:0/134569495 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe270003c70 con 0x7fe27810b930 2024-03-26T06:47:36.794 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-03-26T06:47:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.792+0000 7fe27dcd1640 1 -- 172.21.15.38:0/134569495 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2600a29a0 msgr2=0x7fe2600a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.792+0000 7fe27dcd1640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2600a29a0 0x7fe2600a4e60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe268005e10 tx=0x7fe268005da0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.792+0000 7fe27dcd1640 1 -- 172.21.15.38:0/134569495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe27810b930 msgr2=0x7fe278100c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.792+0000 7fe27dcd1640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe27810b930 0x7fe278100c30 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fe27002c790 tx=0x7fe270035ed0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.792+0000 7fe27dcd1640 1 -- 172.21.15.38:0/134569495 shutdown_connections 2024-03-26T06:47:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.792+0000 7fe27dcd1640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2600a29a0 0x7fe2600a4e60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.792+0000 7fe27dcd1640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe27810b930 0x7fe278100c30 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.792+0000 7fe27dcd1640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe278102e80 0x7fe2781006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.792+0000 7fe27dcd1640 1 --2- 172.21.15.38:0/134569495 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe278102560 0x7fe2781001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.792+0000 7fe27dcd1640 1 -- 172.21.15.38:0/134569495 >> 172.21.15.38:0/134569495 conn(0x7fe278077350 msgr2=0x7fe2780fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.793+0000 7fe27dcd1640 1 -- 172.21.15.38:0/134569495 shutdown_connections 2024-03-26T06:47:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.793+0000 7fe27dcd1640 1 -- 172.21.15.38:0/134569495 wait complete. 2024-03-26T06:47:36.807 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-26T06:47:36.912 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.907+0000 7fee523cd640 1 -- 172.21.15.38:0/4136573249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c06a5c0 msgr2=0x7fee4c105f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.907+0000 7fee523cd640 1 --2- 172.21.15.38:0/4136573249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c06a5c0 0x7fee4c105f30 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fee40009900 tx=0x7fee4002bc30 comp rx=0 tx=0).stop 2024-03-26T06:47:36.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.907+0000 7fee523cd640 1 -- 172.21.15.38:0/4136573249 shutdown_connections 2024-03-26T06:47:36.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.907+0000 7fee523cd640 1 --2- 172.21.15.38:0/4136573249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee4c106610 0x7fee4c10e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.907+0000 7fee523cd640 1 --2- 172.21.15.38:0/4136573249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c06a5c0 0x7fee4c105f30 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.907+0000 7fee523cd640 1 --2- 172.21.15.38:0/4136573249 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee4c069c10 0x7fee4c069ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.907+0000 7fee523cd640 1 -- 172.21.15.38:0/4136573249 >> 172.21.15.38:0/4136573249 conn(0x7fee4c077290 msgr2=0x7fee4c0776a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:36.913 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.907+0000 7fee523cd640 1 -- 172.21.15.38:0/4136573249 shutdown_connections 2024-03-26T06:47:36.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.907+0000 7fee523cd640 1 -- 172.21.15.38:0/4136573249 wait complete. 2024-03-26T06:47:36.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee523cd640 1 Processor -- start 2024-03-26T06:47:36.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee523cd640 1 -- start start 2024-03-26T06:47:36.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee523cd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c069c10 0x7fee4c19d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee523cd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee4c106610 0x7fee4c19ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee523cd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee4c197a20 0x7fee4c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee523cd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c111700 con 0x7fee4c069c10 2024-03-26T06:47:36.914 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee523cd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c111580 con 0x7fee4c197a20 2024-03-26T06:47:36.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee523cd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee4c111880 con 0x7fee4c106610 2024-03-26T06:47:36.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee4bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c069c10 0x7fee4c19d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee5096a640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee4c197a20 0x7fee4c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee4bfff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c069c10 0x7fee4c19d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:51542/0 (socket says 172.21.15.38:51542) 2024-03-26T06:47:36.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee4bfff640 1 -- 172.21.15.38:0/1047302382 learned_addr learned my addr 172.21.15.38:0/1047302382 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:47:36.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee4b7fe640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee4c106610 0x7fee4c19ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.908+0000 7fee4bfff640 1 -- 172.21.15.38:0/1047302382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee4c106610 msgr2=0x7fee4c19ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.909+0000 7fee4bfff640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee4c106610 0x7fee4c19ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.915 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.909+0000 7fee4bfff640 1 -- 172.21.15.38:0/1047302382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee4c197a20 msgr2=0x7fee4c197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:36.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.909+0000 7fee4bfff640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee4c197a20 0x7fee4c197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:36.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.909+0000 7fee4bfff640 1 -- 172.21.15.38:0/1047302382 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee4c198540 con 0x7fee4c069c10 2024-03-26T06:47:36.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.909+0000 7fee5096a640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee4c197a20 0x7fee4c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:47:36.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.909+0000 7fee4bfff640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c069c10 0x7fee4c19d870 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00b4a0 tx=0x7fee3c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:36.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.909+0000 7fee497fa640 1 -- 172.21.15.38:0/1047302382 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3c014070 con 0x7fee4c069c10 2024-03-26T06:47:36.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.909+0000 7fee523cd640 1 -- 172.21.15.38:0/1047302382 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee4c198830 con 0x7fee4c069c10 2024-03-26T06:47:36.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.909+0000 7fee497fa640 1 -- 172.21.15.38:0/1047302382 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee3c010af0 con 0x7fee4c069c10 2024-03-26T06:47:36.916 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.909+0000 7fee497fa640 1 -- 172.21.15.38:0/1047302382 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee3c017d10 con 0x7fee4c069c10 2024-03-26T06:47:36.917 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.909+0000 7fee523cd640 1 -- 172.21.15.38:0/1047302382 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee4c1a4ad0 con 0x7fee4c069c10 2024-03-26T06:47:36.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.910+0000 7fee523cd640 1 -- 172.21.15.38:0/1047302382 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee4c068530 con 0x7fee4c069c10 2024-03-26T06:47:36.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.911+0000 7fee497fa640 1 -- 172.21.15.38:0/1047302382 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fee3c017070 con 0x7fee4c069c10 2024-03-26T06:47:36.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.911+0000 7fee497fa640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fee340a2970 0x7fee340a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:47:36.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.912+0000 7fee4b7fe640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fee340a2970 0x7fee340a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:47:36.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.912+0000 7fee497fa640 1 -- 172.21.15.38:0/1047302382 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8585+0+0 (secure 0 0 0) 0x7fee3c0ce110 con 0x7fee4c069c10 2024-03-26T06:47:36.918 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.912+0000 7fee4b7fe640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fee340a2970 0x7fee340a4e30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0ffd40 tx=0x7fee40005fd0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:47:36.919 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:36.916+0000 7fee497fa640 1 -- 172.21.15.38:0/1047302382 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fee3c09af40 con 0x7fee4c069c10 2024-03-26T06:47:37.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.154+0000 7fee523cd640 1 -- 172.21.15.38:0/1047302382 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7fee4c109860 con 0x7fee4c069c10 2024-03-26T06:47:37.183 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.162+0000 7fee497fa640 1 -- 172.21.15.38:0/1047302382 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) v1 ==== 81+0+740363 (secure 0 0 0) 0x7fee3c09a840 con 0x7fee4c069c10 2024-03-26T06:47:37.184 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:47:37.184 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:47:37.184 INFO:teuthology.orchestra.run.smithi038.stdout: "epoch": 26, 2024-03-26T06:47:37.184 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.184 INFO:teuthology.orchestra.run.smithi038.stdout: "active_gid": 24883, 2024-03-26T06:47:37.184 INFO:teuthology.orchestra.run.smithi038.stdout: "active_name": "x", 2024-03-26T06:47:37.184 INFO:teuthology.orchestra.run.smithi038.stdout: "active_addrs": { 2024-03-26T06:47:37.185 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-26T06:47:37.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.185 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-26T06:47:37.185 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6800", 2024-03-26T06:47:37.185 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1823987770 2024-03-26T06:47:37.185 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.185 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.186 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2024-03-26T06:47:37.186 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:6801", 2024-03-26T06:47:37.186 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1823987770 2024-03-26T06:47:37.186 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.186 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:47:37.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.186 INFO:teuthology.orchestra.run.smithi038.stdout: "active_addr": "172.21.15.38:6801/1823987770", 2024-03-26T06:47:37.187 INFO:teuthology.orchestra.run.smithi038.stdout: "active_change": "2024-03-26T06:37:32.618928+0000", 2024-03-26T06:47:37.187 INFO:teuthology.orchestra.run.smithi038.stdout: "active_mgr_features": 4540701547738038271, 2024-03-26T06:47:37.187 INFO:teuthology.orchestra.run.smithi038.stdout: "available": true, 2024-03-26T06:47:37.187 INFO:teuthology.orchestra.run.smithi038.stdout: "standbys": [ 2024-03-26T06:47:37.187 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.187 INFO:teuthology.orchestra.run.smithi038.stdout: "gid": 24880, 2024-03-26T06:47:37.187 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "z", 2024-03-26T06:47:37.187 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_features": 4540701547738038271, 2024-03-26T06:47:37.188 INFO:teuthology.orchestra.run.smithi038.stdout: "available_modules": [ 2024-03-26T06:47:37.188 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.188 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "alerts", 2024-03-26T06:47:37.188 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.188 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.188 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.188 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:37.188 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:37.189 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.189 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.189 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.189 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:37.189 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.189 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.189 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.189 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How frequently to reexamine health status", 2024-03-26T06:47:37.189 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.190 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.190 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.190 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.190 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.190 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.190 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.190 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.190 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.191 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.191 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.191 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.191 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.191 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.192 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.192 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.192 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.192 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.192 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.192 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.192 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.192 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.193 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.193 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.193 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.193 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.193 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.193 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.193 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.194 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.194 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.194 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.194 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.194 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.194 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.195 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.195 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.195 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.195 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.196 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.196 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.196 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.196 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.196 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.196 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.196 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.196 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.196 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.197 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.197 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.197 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.197 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.197 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.198 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.198 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.198 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.198 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.199 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.199 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.199 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.199 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.199 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.199 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.200 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.201 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.201 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.201 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.201 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.201 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.201 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.202 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_destination": { 2024-03-26T06:47:37.202 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_destination", 2024-03-26T06:47:37.202 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.202 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.202 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.202 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.202 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.203 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.203 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.203 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email address to send alerts to", 2024-03-26T06:47:37.203 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.203 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.203 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.203 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_from_name": { 2024-03-26T06:47:37.204 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_from_name", 2024-03-26T06:47:37.204 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.204 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.204 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.204 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "Ceph", 2024-03-26T06:47:37.204 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.204 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.204 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.204 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email From: name", 2024-03-26T06:47:37.205 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.205 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.205 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.205 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_host": { 2024-03-26T06:47:37.205 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_host", 2024-03-26T06:47:37.205 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.206 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.206 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.206 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.206 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.206 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.206 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.206 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP server", 2024-03-26T06:47:37.206 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.207 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.207 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.207 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_password": { 2024-03-26T06:47:37.207 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_password", 2024-03-26T06:47:37.207 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.207 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.207 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.208 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.208 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.208 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.208 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.208 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Password to authenticate with", 2024-03-26T06:47:37.208 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.208 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.209 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.209 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.209 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_port": { 2024-03-26T06:47:37.209 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_port", 2024-03-26T06:47:37.209 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.209 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.209 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.209 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "465", 2024-03-26T06:47:37.209 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.210 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.210 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.210 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP port", 2024-03-26T06:47:37.210 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.210 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.210 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.210 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_sender": { 2024-03-26T06:47:37.211 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_sender", 2024-03-26T06:47:37.211 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.211 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.211 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.211 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.211 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.211 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.211 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.212 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP envelope sender", 2024-03-26T06:47:37.212 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.212 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.212 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.212 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_ssl": { 2024-03-26T06:47:37.212 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_ssl", 2024-03-26T06:47:37.212 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.213 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.213 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.213 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.213 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.213 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.213 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.213 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-26T06:47:37.213 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.214 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.214 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.214 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_user": { 2024-03-26T06:47:37.214 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_user", 2024-03-26T06:47:37.214 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.214 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.214 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.215 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.215 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.215 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.215 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.215 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "User to authenticate as", 2024-03-26T06:47:37.215 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.215 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.215 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.216 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "balancer", 2024-03-26T06:47:37.216 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.216 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.217 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.217 INFO:teuthology.orchestra.run.smithi038.stdout: "active": { 2024-03-26T06:47:37.217 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "active", 2024-03-26T06:47:37.217 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.217 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.217 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.217 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.217 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically balance PGs across cluster", 2024-03-26T06:47:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.218 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_time": { 2024-03-26T06:47:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_time", 2024-03-26T06:47:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0000", 2024-03-26T06:47:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.219 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "beginning time of day to automatically balance", 2024-03-26T06:47:37.220 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-26T06:47:37.220 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.220 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.220 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.220 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_weekday": { 2024-03-26T06:47:37.220 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_weekday", 2024-03-26T06:47:37.220 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:37.220 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.221 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.221 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:37.221 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-03-26T06:47:37.221 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-03-26T06:47:37.221 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.221 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-26T06:47:37.221 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-26T06:47:37.221 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.222 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.222 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.222 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_max_iterations": { 2024-03-26T06:47:37.222 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_max_iterations", 2024-03-26T06:47:37.222 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:37.222 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.222 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.222 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "25", 2024-03-26T06:47:37.223 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:37.223 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "250", 2024-03-26T06:47:37.223 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.223 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-26T06:47:37.223 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.223 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.223 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.223 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_metrics": { 2024-03-26T06:47:37.224 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_metrics", 2024-03-26T06:47:37.224 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.224 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.224 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.224 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "pgs,objects,bytes", 2024-03-26T06:47:37.224 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.224 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.224 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.225 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-26T06:47:37.225 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-26T06:47:37.225 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.225 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.225 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_step": { 2024-03-26T06:47:37.225 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_step", 2024-03-26T06:47:37.225 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:37.226 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.226 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.226 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.5", 2024-03-26T06:47:37.226 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0.001", 2024-03-26T06:47:37.226 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "0.999", 2024-03-26T06:47:37.226 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.226 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "aggressiveness of optimization", 2024-03-26T06:47:37.226 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-26T06:47:37.227 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.227 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.227 INFO:teuthology.orchestra.run.smithi038.stdout: "end_time": { 2024-03-26T06:47:37.227 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_time", 2024-03-26T06:47:37.227 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.227 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.228 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.228 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2359", 2024-03-26T06:47:37.228 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.228 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.228 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.228 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ending time of day to automatically balance", 2024-03-26T06:47:37.228 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-26T06:47:37.228 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.229 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.229 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.229 INFO:teuthology.orchestra.run.smithi038.stdout: "end_weekday": { 2024-03-26T06:47:37.229 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_weekday", 2024-03-26T06:47:37.229 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:37.229 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.229 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.229 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:37.229 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-03-26T06:47:37.230 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-03-26T06:47:37.230 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.230 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-26T06:47:37.230 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-26T06:47:37.230 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.230 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.230 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.231 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.231 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.231 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.231 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.231 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.231 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.231 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.231 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.231 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.232 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.232 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.232 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.232 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.232 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.232 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.232 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.233 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.233 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.233 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.233 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.233 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.233 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.233 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.234 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.234 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.234 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.234 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.234 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.234 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.234 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.234 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.235 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.235 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.235 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.235 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.235 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.235 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.235 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.236 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.236 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.236 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.236 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.236 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.236 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.236 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.236 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.237 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.237 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.237 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.237 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.237 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.237 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.237 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.237 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.238 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.238 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.238 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.238 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.238 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.238 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.238 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.239 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.239 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.239 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.239 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.239 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.239 INFO:teuthology.orchestra.run.smithi038.stdout: "min_score": { 2024-03-26T06:47:37.239 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_score", 2024-03-26T06:47:37.240 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:37.240 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.240 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.240 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:37.240 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.240 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.240 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.240 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-26T06:47:37.241 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.241 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.241 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.241 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-03-26T06:47:37.241 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-03-26T06:47:37.241 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.241 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.241 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.242 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "upmap", 2024-03-26T06:47:37.242 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.242 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.242 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.242 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-compat", 2024-03-26T06:47:37.242 INFO:teuthology.orchestra.run.smithi038.stdout: "none", 2024-03-26T06:47:37.242 INFO:teuthology.orchestra.run.smithi038.stdout: "read", 2024-03-26T06:47:37.242 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap", 2024-03-26T06:47:37.243 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap-read" 2024-03-26T06:47:37.243 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.243 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Balancer mode", 2024-03-26T06:47:37.243 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.243 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.243 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.243 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_ids": { 2024-03-26T06:47:37.244 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_ids", 2024-03-26T06:47:37.244 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.244 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.244 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.244 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.244 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.244 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.244 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.245 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-26T06:47:37.245 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.245 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.245 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.245 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.245 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:37.245 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:37.245 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.245 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.246 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.246 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:37.246 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.246 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.246 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.246 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-26T06:47:37.246 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.246 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.247 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.247 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.247 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_deviation": { 2024-03-26T06:47:37.247 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_deviation", 2024-03-26T06:47:37.247 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.247 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.247 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.247 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-26T06:47:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-26T06:47:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_optimizations": { 2024-03-26T06:47:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_optimizations", 2024-03-26T06:47:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.249 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-26T06:47:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.250 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm", 2024-03-26T06:47:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_down_multiplier": { 2024-03-26T06:47:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_down_multiplier", 2024-03-26T06:47:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:37.251 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-03-26T06:47:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-26T06:47:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.252 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_refresh_rate": { 2024-03-26T06:47:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_refresh_rate", 2024-03-26T06:47:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "20", 2024-03-26T06:47:37.253 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-26T06:47:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.254 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_starting_port": { 2024-03-26T06:47:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_starting_port", 2024-03-26T06:47:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4721", 2024-03-26T06:47:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.255 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-26T06:47:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_ptrace": { 2024-03-26T06:47:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_ptrace", 2024-03-26T06:47:37.256 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.257 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.257 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.257 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.257 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.257 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.257 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.257 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-26T06:47:37.257 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-26T06:47:37.258 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.258 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.258 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_interval": { 2024-03-26T06:47:37.258 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_interval", 2024-03-26T06:47:37.258 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.258 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.259 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.259 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:37.259 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.259 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.259 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.259 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-26T06:47:37.259 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.259 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.260 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.260 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_memory_target_ratio": { 2024-03-26T06:47:37.260 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_memory_target_ratio", 2024-03-26T06:47:37.260 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:37.260 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.260 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.260 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.7", 2024-03-26T06:47:37.260 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.261 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.261 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.261 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-26T06:47:37.261 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.261 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.261 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.261 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_log_destination": { 2024-03-26T06:47:37.262 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm_log_destination", 2024-03-26T06:47:37.262 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.262 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.262 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.262 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.262 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.262 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.262 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.263 INFO:teuthology.orchestra.run.smithi038.stdout: "file", 2024-03-26T06:47:37.263 INFO:teuthology.orchestra.run.smithi038.stdout: "file,syslog", 2024-03-26T06:47:37.263 INFO:teuthology.orchestra.run.smithi038.stdout: "syslog" 2024-03-26T06:47:37.263 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.263 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-26T06:47:37.263 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.263 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.263 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.264 INFO:teuthology.orchestra.run.smithi038.stdout: "cgroups_split": { 2024-03-26T06:47:37.264 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cgroups_split", 2024-03-26T06:47:37.264 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.264 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.264 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.264 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.264 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.264 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.265 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.265 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-26T06:47:37.265 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.265 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.265 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.265 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.265 INFO:teuthology.orchestra.run.smithi038.stdout: "config_checks_enabled": { 2024-03-26T06:47:37.265 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_checks_enabled", 2024-03-26T06:47:37.266 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.266 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.266 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.266 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.266 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.266 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.266 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.266 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-26T06:47:37.267 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.267 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.267 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.267 INFO:teuthology.orchestra.run.smithi038.stdout: "config_dashboard": { 2024-03-26T06:47:37.267 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_dashboard", 2024-03-26T06:47:37.267 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.267 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.268 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.268 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.268 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.268 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.268 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.268 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-26T06:47:37.268 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.268 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.268 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.269 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_alertmanager": { 2024-03-26T06:47:37.269 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_alertmanager", 2024-03-26T06:47:37.269 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.269 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.269 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.269 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-26T06:47:37.269 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.270 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.270 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.270 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:37.270 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.270 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.270 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.270 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_base": { 2024-03-26T06:47:37.271 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_base", 2024-03-26T06:47:37.271 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.271 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.271 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.271 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-26T06:47:37.271 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.271 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.271 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.272 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Container image name, without the tag", 2024-03-26T06:47:37.272 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.272 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.272 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.272 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_elasticsearch": { 2024-03-26T06:47:37.272 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_elasticsearch", 2024-03-26T06:47:37.272 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.272 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.273 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.273 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-26T06:47:37.273 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.273 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.273 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.273 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "elasticsearch container image", 2024-03-26T06:47:37.273 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.273 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.274 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.274 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_grafana": { 2024-03-26T06:47:37.274 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_grafana", 2024-03-26T06:47:37.274 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.274 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.274 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.274 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-26T06:47:37.275 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.275 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.275 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.275 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:37.275 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.275 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.275 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.275 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.276 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_haproxy": { 2024-03-26T06:47:37.276 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_haproxy", 2024-03-26T06:47:37.276 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.276 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.276 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.276 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-26T06:47:37.276 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.276 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.277 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.277 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "HAproxy container image", 2024-03-26T06:47:37.277 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.277 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.277 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.277 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.277 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_agent": { 2024-03-26T06:47:37.278 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_agent", 2024-03-26T06:47:37.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.278 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.278 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.278 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-26T06:47:37.278 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.278 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.278 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.279 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger agent container image", 2024-03-26T06:47:37.279 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.279 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.279 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.279 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.279 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_collector": { 2024-03-26T06:47:37.279 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_collector", 2024-03-26T06:47:37.279 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.280 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.280 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.280 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-26T06:47:37.280 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.280 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.280 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.280 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger collector container image", 2024-03-26T06:47:37.280 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.281 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.281 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.281 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_query": { 2024-03-26T06:47:37.281 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_query", 2024-03-26T06:47:37.281 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.281 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.281 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.281 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-26T06:47:37.282 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.282 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.282 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.282 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger query container image", 2024-03-26T06:47:37.282 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.282 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.282 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.283 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_keepalived": { 2024-03-26T06:47:37.283 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_keepalived", 2024-03-26T06:47:37.283 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.283 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.283 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.283 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-26T06:47:37.283 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.283 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.284 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.284 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Keepalived container image", 2024-03-26T06:47:37.284 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.284 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.284 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.284 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.284 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_loki": { 2024-03-26T06:47:37.284 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_loki", 2024-03-26T06:47:37.285 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.285 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.285 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.285 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-26T06:47:37.285 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.285 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.285 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.285 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Loki container image", 2024-03-26T06:47:37.286 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.286 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.286 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.286 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.286 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_node_exporter": { 2024-03-26T06:47:37.286 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_node_exporter", 2024-03-26T06:47:37.286 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.286 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.286 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.287 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-26T06:47:37.287 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.287 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.287 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.287 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:37.287 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.287 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.288 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.288 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_nvmeof": { 2024-03-26T06:47:37.288 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_nvmeof", 2024-03-26T06:47:37.288 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.288 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.288 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.288 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-26T06:47:37.289 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.289 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.289 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.289 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Nvme-of container image", 2024-03-26T06:47:37.289 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.289 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.289 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.289 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_prometheus": { 2024-03-26T06:47:37.290 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_prometheus", 2024-03-26T06:47:37.290 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.290 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.290 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.290 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-26T06:47:37.290 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.290 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.290 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.291 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:37.291 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.291 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.291 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.291 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_promtail": { 2024-03-26T06:47:37.291 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_promtail", 2024-03-26T06:47:37.291 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.291 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.292 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.292 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-26T06:47:37.292 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.292 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.292 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.292 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Promtail container image", 2024-03-26T06:47:37.292 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.292 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.293 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.293 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_snmp_gateway": { 2024-03-26T06:47:37.293 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_snmp_gateway", 2024-03-26T06:47:37.293 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.293 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.293 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.293 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-26T06:47:37.294 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.294 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.294 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.294 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SNMP Gateway container image", 2024-03-26T06:47:37.294 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.294 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.294 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.295 INFO:teuthology.orchestra.run.smithi038.stdout: "container_init": { 2024-03-26T06:47:37.295 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_init", 2024-03-26T06:47:37.295 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.295 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.295 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.295 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.295 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.295 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.295 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.296 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Run podman/docker with `--init`", 2024-03-26T06:47:37.296 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.296 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.296 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.296 INFO:teuthology.orchestra.run.smithi038.stdout: "daemon_cache_timeout": { 2024-03-26T06:47:37.296 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "daemon_cache_timeout", 2024-03-26T06:47:37.296 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.297 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.297 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.297 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:37.297 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.297 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.297 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.297 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-26T06:47:37.298 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.298 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.298 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.298 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.298 INFO:teuthology.orchestra.run.smithi038.stdout: "default_cephadm_command_timeout": { 2024-03-26T06:47:37.298 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_cephadm_command_timeout", 2024-03-26T06:47:37.298 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.299 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.299 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.299 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "900", 2024-03-26T06:47:37.299 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.299 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.299 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.299 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-26T06:47:37.299 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.300 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.300 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.300 INFO:teuthology.orchestra.run.smithi038.stdout: "default_registry": { 2024-03-26T06:47:37.300 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_registry", 2024-03-26T06:47:37.300 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.300 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.300 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.300 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io", 2024-03-26T06:47:37.301 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.301 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.301 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.301 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-26T06:47:37.301 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.301 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.301 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.302 INFO:teuthology.orchestra.run.smithi038.stdout: "device_cache_timeout": { 2024-03-26T06:47:37.302 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_cache_timeout", 2024-03-26T06:47:37.302 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.302 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.302 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.302 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1800", 2024-03-26T06:47:37.302 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.302 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache device inventory", 2024-03-26T06:47:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "device_enhanced_scan": { 2024-03-26T06:47:37.303 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_enhanced_scan", 2024-03-26T06:47:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.304 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-26T06:47:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "facts_cache_timeout": { 2024-03-26T06:47:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "facts_cache_timeout", 2024-03-26T06:47:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.305 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache host facts data", 2024-03-26T06:47:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.306 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "host_check_interval": { 2024-03-26T06:47:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "host_check_interval", 2024-03-26T06:47:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.307 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to perform a host check", 2024-03-26T06:47:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "hw_monitoring": { 2024-03-26T06:47:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hw_monitoring", 2024-03-26T06:47:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.309 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-26T06:47:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: "inventory_list_all": { 2024-03-26T06:47:37.310 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "inventory_list_all", 2024-03-26T06:47:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.311 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-26T06:47:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.312 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.313 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.313 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.313 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.313 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.313 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.313 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.313 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.313 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.314 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.314 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.314 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.314 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.314 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.314 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.315 INFO:teuthology.orchestra.run.smithi038.stdout: "log_refresh_metadata": { 2024-03-26T06:47:37.315 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_refresh_metadata", 2024-03-26T06:47:37.315 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.315 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.315 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.315 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.315 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.315 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.316 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.316 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-26T06:47:37.316 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.316 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.316 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.316 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.316 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.317 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.317 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.317 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.317 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.317 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.317 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.317 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.317 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-26T06:47:37.318 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.318 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.318 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.318 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.318 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.318 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.318 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.319 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.319 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.319 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.319 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.319 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.319 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.319 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.320 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.320 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.320 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.320 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.320 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.320 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.320 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.320 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.321 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.321 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.321 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.321 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.321 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.321 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.321 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.322 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.322 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.322 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.322 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.322 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.322 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.322 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.322 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-26T06:47:37.323 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-26T06:47:37.323 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.323 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.323 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.323 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.323 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.323 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.323 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.324 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-26T06:47:37.324 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.324 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.324 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.324 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-26T06:47:37.324 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-26T06:47:37.324 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.324 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.325 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.325 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "*", 2024-03-26T06:47:37.325 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.325 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.325 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.325 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-26T06:47:37.325 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.326 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.326 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.326 INFO:teuthology.orchestra.run.smithi038.stdout: "max_count_per_host": { 2024-03-26T06:47:37.326 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_count_per_host", 2024-03-26T06:47:37.326 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.326 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.326 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.326 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:37.327 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.327 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.327 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.327 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of daemons per service per host", 2024-03-26T06:47:37.327 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.327 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.327 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.328 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd_draining_count": { 2024-03-26T06:47:37.328 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_osd_draining_count", 2024-03-26T06:47:37.328 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.328 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.328 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.328 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:37.328 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.328 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.329 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.329 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-26T06:47:37.329 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.329 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.329 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.329 INFO:teuthology.orchestra.run.smithi038.stdout: "migration_current": { 2024-03-26T06:47:37.329 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "migration_current", 2024-03-26T06:47:37.329 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.330 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.330 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.330 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.330 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.330 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.330 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.330 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "internal - do not modify", 2024-03-26T06:47:37.330 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.331 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.331 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.331 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-03-26T06:47:37.331 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-03-26T06:47:37.331 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.331 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.331 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.332 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "root", 2024-03-26T06:47:37.332 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.332 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.332 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.332 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm-package", 2024-03-26T06:47:37.332 INFO:teuthology.orchestra.run.smithi038.stdout: "root" 2024-03-26T06:47:37.332 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.332 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "mode for remote execution of cephadm", 2024-03-26T06:47:37.333 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.333 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.333 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.333 INFO:teuthology.orchestra.run.smithi038.stdout: "oob_default_addr": { 2024-03-26T06:47:37.333 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "oob_default_addr", 2024-03-26T06:47:37.333 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.333 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.334 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.334 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "169.254.1.1", 2024-03-26T06:47:37.334 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.334 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.334 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.334 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-26T06:47:37.334 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.334 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.334 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.335 INFO:teuthology.orchestra.run.smithi038.stdout: "prometheus_alerts_path": { 2024-03-26T06:47:37.335 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus_alerts_path", 2024-03-26T06:47:37.335 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.335 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.335 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.335 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-26T06:47:37.336 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.336 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.336 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.336 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-26T06:47:37.336 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.337 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.337 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.337 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_insecure": { 2024-03-26T06:47:37.337 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_insecure", 2024-03-26T06:47:37.337 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.337 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.337 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.338 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.338 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.338 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.338 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.338 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-26T06:47:37.338 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.338 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.338 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.339 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.339 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_password": { 2024-03-26T06:47:37.339 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_password", 2024-03-26T06:47:37.339 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.339 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.339 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.339 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.339 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.340 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.340 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.340 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-26T06:47:37.340 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.340 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.340 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.340 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_url": { 2024-03-26T06:47:37.341 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_url", 2024-03-26T06:47:37.341 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.341 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.341 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.341 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.341 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.341 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.341 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.341 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-26T06:47:37.342 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.342 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.342 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.342 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_username": { 2024-03-26T06:47:37.342 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_username", 2024-03-26T06:47:37.342 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.342 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.343 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.343 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.343 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.343 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.343 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.343 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-26T06:47:37.343 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.343 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.344 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.344 INFO:teuthology.orchestra.run.smithi038.stdout: "secure_monitoring_stack": { 2024-03-26T06:47:37.344 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "secure_monitoring_stack", 2024-03-26T06:47:37.344 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.344 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.344 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.344 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.345 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.345 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.345 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.345 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-26T06:47:37.345 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.345 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.345 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.345 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "service_discovery_port": { 2024-03-26T06:47:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "service_discovery_port", 2024-03-26T06:47:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8765", 2024-03-26T06:47:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.346 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "cephadm service discovery port", 2024-03-26T06:47:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "ssh_config_file": { 2024-03-26T06:47:37.347 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssh_config_file", 2024-03-26T06:47:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.348 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-26T06:47:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "use_agent": { 2024-03-26T06:47:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_agent", 2024-03-26T06:47:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.349 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-26T06:47:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.350 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "use_repo_digest": { 2024-03-26T06:47:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_repo_digest", 2024-03-26T06:47:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.351 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-26T06:47:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.352 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_failed_host_check": { 2024-03-26T06:47:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_failed_host_check", 2024-03-26T06:47:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.353 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if the host check fails", 2024-03-26T06:47:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_daemons": { 2024-03-26T06:47:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_daemons", 2024-03-26T06:47:37.354 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.355 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.355 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.355 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.355 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.355 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.355 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.355 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-26T06:47:37.355 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.355 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.356 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.356 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_hosts": { 2024-03-26T06:47:37.356 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_hosts", 2024-03-26T06:47:37.356 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.356 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.356 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.356 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.357 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.357 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.357 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.357 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-26T06:47:37.357 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.357 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.357 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.358 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.358 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.358 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.358 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crash", 2024-03-26T06:47:37.358 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.358 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.358 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.359 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.359 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.359 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.359 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.359 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.359 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.359 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.359 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.359 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.360 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.360 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.360 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.360 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.360 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.360 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.360 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.360 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.361 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.361 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.361 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.361 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.361 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.361 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.361 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.362 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.362 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.362 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.362 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.362 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.362 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.368 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.368 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.368 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.369 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.370 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.371 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.372 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "retain_interval": { 2024-03-26T06:47:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retain_interval", 2024-03-26T06:47:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.373 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "31536000", 2024-03-26T06:47:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-26T06:47:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.374 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_recent_interval": { 2024-03-26T06:47:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_recent_interval", 2024-03-26T06:47:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1209600", 2024-03-26T06:47:37.375 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-26T06:47:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dashboard", 2024-03-26T06:47:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-26T06:47:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-26T06:47:37.377 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.378 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_HOST": { 2024-03-26T06:47:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-26T06:47:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.379 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.380 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-26T06:47:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-26T06:47:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.381 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_ENABLED": { 2024-03-26T06:47:37.382 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_ENABLED", 2024-03-26T06:47:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.383 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.384 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.384 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.384 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.384 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-26T06:47:37.385 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-26T06:47:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.386 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "ENABLE_BROWSABLE_API": { 2024-03-26T06:47:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-26T06:47:37.387 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.388 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-26T06:47:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-26T06:47:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.389 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.390 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-26T06:47:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-26T06:47:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.391 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.392 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-26T06:47:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-26T06:47:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.393 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-26T06:47:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-26T06:47:37.394 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.395 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-26T06:47:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-26T06:47:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.396 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.397 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-26T06:47:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-26T06:47:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.398 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-26T06:47:37.399 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-26T06:47:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.400 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-26T06:47:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-26T06:47:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.401 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.402 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_PASSWORD": { 2024-03-26T06:47:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-26T06:47:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.403 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-03-26T06:47:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.404 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-26T06:47:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-26T06:47:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.405 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_URL": { 2024-03-26T06:47:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_URL", 2024-03-26T06:47:37.406 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.407 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_USERNAME": { 2024-03-26T06:47:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-26T06:47:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.408 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-03-26T06:47:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-26T06:47:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-26T06:47:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.410 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-26T06:47:37.411 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-26T06:47:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.412 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-26T06:47:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-26T06:47:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.413 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.414 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-26T06:47:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-26T06:47:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.415 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.416 INFO:teuthology.orchestra.run.smithi038.stdout: "MULTICLUSTER_CONFIG": { 2024-03-26T06:47:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-26T06:47:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "{}", 2024-03-26T06:47:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.417 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_HOST": { 2024-03-26T06:47:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-26T06:47:37.418 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.419 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-26T06:47:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-26T06:47:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.420 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.421 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-26T06:47:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-26T06:47:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.422 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-26T06:47:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-26T06:47:37.423 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.424 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-26T06:47:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-26T06:47:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.425 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.426 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-26T06:47:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-26T06:47:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.427 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-26T06:47:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-26T06:47:37.428 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.429 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-26T06:47:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-26T06:47:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.430 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.431 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-26T06:47:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-26T06:47:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.432 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_ENABLED": { 2024-03-26T06:47:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-26T06:47:37.433 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.434 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.435 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.437 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.438 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-26T06:47:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-26T06:47:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8", 2024-03-26T06:47:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.439 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-26T06:47:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-26T06:47:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.440 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "45", 2024-03-26T06:47:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.441 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ACCESS_KEY": { 2024-03-26T06:47:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-26T06:47:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.442 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.443 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-26T06:47:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-26T06:47:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-03-26T06:47:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.444 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SECRET_KEY": { 2024-03-26T06:47:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-26T06:47:37.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.446 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SSL_VERIFY": { 2024-03-26T06:47:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-26T06:47:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.447 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.448 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "UNSAFE_TLS_v1_2": { 2024-03-26T06:47:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-26T06:47:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.449 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.450 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-26T06:47:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-26T06:47:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.451 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-26T06:47:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-26T06:47:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.452 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.453 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-26T06:47:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-26T06:47:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:37.454 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "cross_origin_url": { 2024-03-26T06:47:37.455 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cross_origin_url", 2024-03-26T06:47:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.456 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "crt_file": { 2024-03-26T06:47:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crt_file", 2024-03-26T06:47:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.457 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.458 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "debug": { 2024-03-26T06:47:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "debug", 2024-03-26T06:47:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.459 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable/disable debug options", 2024-03-26T06:47:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.460 INFO:teuthology.orchestra.run.smithi038.stdout: "jwt_token_ttl": { 2024-03-26T06:47:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "jwt_token_ttl", 2024-03-26T06:47:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "28800", 2024-03-26T06:47:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.461 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-03-26T06:47:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-03-26T06:47:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.462 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.463 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.464 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.465 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.466 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.467 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.468 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.469 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.470 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.471 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "motd": { 2024-03-26T06:47:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "motd", 2024-03-26T06:47:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.472 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "The message of the day", 2024-03-26T06:47:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.473 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.474 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.474 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_resolve_ip_addr": { 2024-03-26T06:47:37.474 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "redirect_resolve_ip_addr", 2024-03-26T06:47:37.474 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.474 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.474 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.474 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.474 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.475 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.475 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.475 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.475 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.475 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.475 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.475 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-03-26T06:47:37.475 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-03-26T06:47:37.476 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.476 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.476 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.476 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-03-26T06:47:37.476 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.476 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.476 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.476 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.477 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.477 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.477 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.477 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-03-26T06:47:37.477 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-03-26T06:47:37.477 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.477 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.478 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.478 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8080", 2024-03-26T06:47:37.478 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.478 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.478 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.478 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.478 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.478 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.479 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.479 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-03-26T06:47:37.479 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-03-26T06:47:37.479 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.479 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.479 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.479 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.480 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.480 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.480 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.480 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.480 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.480 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.480 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.480 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.481 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl_server_port": { 2024-03-26T06:47:37.481 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl_server_port", 2024-03-26T06:47:37.481 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.481 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.481 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.481 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8443", 2024-03-26T06:47:37.481 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.481 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.481 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.482 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.482 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.482 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.482 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.482 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.482 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-03-26T06:47:37.482 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-03-26T06:47:37.483 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.483 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.483 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.483 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "redirect", 2024-03-26T06:47:37.483 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.483 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.483 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.483 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.483 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect" 2024-03-26T06:47:37.484 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.484 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.484 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.484 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.484 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.484 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.484 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-03-26T06:47:37.484 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-03-26T06:47:37.485 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.485 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.485 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.485 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-03-26T06:47:37.485 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-03-26T06:47:37.485 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-03-26T06:47:37.485 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.485 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.485 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.486 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.486 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.486 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.486 INFO:teuthology.orchestra.run.smithi038.stdout: "url_prefix": { 2024-03-26T06:47:37.486 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url_prefix", 2024-03-26T06:47:37.486 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.486 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.486 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.487 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.487 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.487 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.487 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.487 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.487 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.487 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.487 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.488 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.488 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.488 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.488 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.488 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-03-26T06:47:37.488 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.488 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.488 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.489 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_monitoring": { 2024-03-26T06:47:37.489 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_monitoring", 2024-03-26T06:47:37.489 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.489 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.489 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.489 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.489 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.489 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.489 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.490 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "monitor device health metrics", 2024-03-26T06:47:37.490 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.490 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.490 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.490 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.490 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.490 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.490 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.491 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.491 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.491 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.491 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.491 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.491 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.491 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.491 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.492 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.492 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.492 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.492 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.492 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.492 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.492 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.492 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.492 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.493 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.493 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.493 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.493 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.493 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.493 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.493 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.493 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.493 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.494 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.494 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.494 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.494 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.494 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.494 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.494 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.495 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.495 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.495 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.495 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.495 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.495 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.495 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.495 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.496 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.496 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.496 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.496 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.496 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.496 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.496 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.496 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.497 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.497 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.497 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.497 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.497 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.497 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.497 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.497 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.497 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.498 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.498 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.498 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.498 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.498 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.498 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.498 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.499 INFO:teuthology.orchestra.run.smithi038.stdout: "mark_out_threshold": { 2024-03-26T06:47:37.499 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mark_out_threshold", 2024-03-26T06:47:37.499 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.499 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.499 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.499 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2419200", 2024-03-26T06:47:37.499 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.499 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.500 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.500 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-26T06:47:37.500 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.500 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.500 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.500 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": { 2024-03-26T06:47:37.500 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_name", 2024-03-26T06:47:37.500 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.501 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.501 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.501 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "device_health_metrics", 2024-03-26T06:47:37.501 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.501 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.501 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.501 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-26T06:47:37.501 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.502 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.502 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.502 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.502 INFO:teuthology.orchestra.run.smithi038.stdout: "retention_period": { 2024-03-26T06:47:37.502 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retention_period", 2024-03-26T06:47:37.502 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.502 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.502 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.502 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15552000", 2024-03-26T06:47:37.503 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.503 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.503 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.503 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain device health metrics", 2024-03-26T06:47:37.503 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.503 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.503 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.504 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_frequency": { 2024-03-26T06:47:37.504 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_frequency", 2024-03-26T06:47:37.504 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.504 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.504 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.504 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-03-26T06:47:37.504 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.504 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.504 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.505 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-26T06:47:37.505 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.505 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.505 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.505 INFO:teuthology.orchestra.run.smithi038.stdout: "self_heal": { 2024-03-26T06:47:37.505 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "self_heal", 2024-03-26T06:47:37.505 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.506 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.506 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.506 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.506 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.506 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.506 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.506 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-26T06:47:37.506 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.507 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.507 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.507 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.507 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:37.507 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:37.507 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.507 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.507 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.508 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:37.508 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.508 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.508 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.508 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and check device health", 2024-03-26T06:47:37.508 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.508 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.508 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.509 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_threshold": { 2024-03-26T06:47:37.509 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_threshold", 2024-03-26T06:47:37.509 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.509 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.509 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.509 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7257600", 2024-03-26T06:47:37.509 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.509 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.510 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.510 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-26T06:47:37.510 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.510 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.510 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.510 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.510 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.510 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.511 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "diskprediction_local", 2024-03-26T06:47:37.511 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.511 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.511 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.511 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.511 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.511 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.511 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.512 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.512 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.512 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.512 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.512 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.512 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.512 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.512 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.513 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.513 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.513 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.513 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.513 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.513 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.513 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.513 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.514 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.514 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.514 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.514 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.514 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.514 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.514 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.515 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.515 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.515 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.515 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.515 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.515 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.515 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.515 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.515 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.516 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.516 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.516 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.516 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.516 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.516 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.516 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.517 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.517 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.517 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.517 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.517 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.517 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.517 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.517 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.518 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.518 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.518 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.518 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.518 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.518 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.518 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.518 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.518 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.519 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.519 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.519 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.519 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.519 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.519 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.519 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.520 INFO:teuthology.orchestra.run.smithi038.stdout: "predict_interval": { 2024-03-26T06:47:37.520 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predict_interval", 2024-03-26T06:47:37.520 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.520 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.520 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.520 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-03-26T06:47:37.520 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.520 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.521 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.521 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.521 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.521 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.521 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.521 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.521 INFO:teuthology.orchestra.run.smithi038.stdout: "predictor_model": { 2024-03-26T06:47:37.521 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predictor_model", 2024-03-26T06:47:37.521 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.522 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.522 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.522 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "prophetstor", 2024-03-26T06:47:37.522 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.522 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.522 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.522 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.522 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.523 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.523 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.523 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.523 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:37.523 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:37.523 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.523 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.523 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.523 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:37.524 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.524 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.524 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.524 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.524 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.524 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.524 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.524 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.525 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.525 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.525 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "influx", 2024-03-26T06:47:37.525 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": false, 2024-03-26T06:47:37.525 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "influxdb python module not found", 2024-03-26T06:47:37.525 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.525 INFO:teuthology.orchestra.run.smithi038.stdout: "batch_size": { 2024-03-26T06:47:37.525 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "batch_size", 2024-03-26T06:47:37.526 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.526 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.526 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.526 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5000", 2024-03-26T06:47:37.526 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.526 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.526 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.526 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-26T06:47:37.527 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.527 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.527 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.527 INFO:teuthology.orchestra.run.smithi038.stdout: "database": { 2024-03-26T06:47:37.527 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "database", 2024-03-26T06:47:37.527 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.527 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.528 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.528 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "ceph", 2024-03-26T06:47:37.528 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.528 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.528 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.528 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-26T06:47:37.528 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.528 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.529 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.529 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": { 2024-03-26T06:47:37.529 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hostname", 2024-03-26T06:47:37.529 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.529 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.529 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.529 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.530 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.530 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.530 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.530 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server hostname", 2024-03-26T06:47:37.530 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.530 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.530 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.531 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:37.531 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:37.531 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.531 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.531 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.531 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "30", 2024-03-26T06:47:37.531 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "5", 2024-03-26T06:47:37.531 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.531 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.532 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-26T06:47:37.532 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.532 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.532 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.532 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.532 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.532 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.533 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.533 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.533 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.533 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.533 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.533 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.533 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.533 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.534 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.534 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.534 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.534 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.534 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.534 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.534 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.534 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.535 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.535 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.535 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.535 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.535 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.535 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.535 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.535 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.536 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.536 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.536 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.536 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.536 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.536 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.536 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.536 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.536 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.537 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.537 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.537 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.537 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.537 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.537 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.537 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.537 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.538 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.538 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.538 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.538 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.538 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.538 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.538 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.538 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.539 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.539 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.539 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.539 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.539 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.539 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.539 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.539 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.540 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.540 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.540 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.540 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.540 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.540 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.540 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.540 INFO:teuthology.orchestra.run.smithi038.stdout: "password": { 2024-03-26T06:47:37.541 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "password", 2024-03-26T06:47:37.541 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.541 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.541 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.541 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.541 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.541 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.541 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.541 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "password of InfluxDB server user", 2024-03-26T06:47:37.542 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.542 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.542 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.542 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.542 INFO:teuthology.orchestra.run.smithi038.stdout: "port": { 2024-03-26T06:47:37.542 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "port", 2024-03-26T06:47:37.542 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.542 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8086", 2024-03-26T06:47:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server port", 2024-03-26T06:47:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.543 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-03-26T06:47:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-03-26T06:47:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "false", 2024-03-26T06:47:37.544 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-26T06:47:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.545 INFO:teuthology.orchestra.run.smithi038.stdout: "threads": { 2024-03-26T06:47:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threads", 2024-03-26T06:47:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "32", 2024-03-26T06:47:37.546 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-26T06:47:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: "username": { 2024-03-26T06:47:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "username", 2024-03-26T06:47:37.547 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "username of InfluxDB server user", 2024-03-26T06:47:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.548 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: "verify_ssl": { 2024-03-26T06:47:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "verify_ssl", 2024-03-26T06:47:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.549 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "true", 2024-03-26T06:47:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-26T06:47:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.550 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "insights", 2024-03-26T06:47:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.551 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.552 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.552 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.552 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.552 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.552 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.552 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.552 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.552 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.552 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.553 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.553 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.553 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.553 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.553 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.553 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.553 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.553 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.553 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.554 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.554 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.554 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.554 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.554 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.554 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.554 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.554 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.555 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.556 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.556 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.556 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.556 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.556 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.556 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.556 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.556 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.556 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.557 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.557 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.557 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.557 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.557 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.557 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.557 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.557 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.558 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.558 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.558 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.558 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.558 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.558 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.558 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.559 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.559 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.559 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.559 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.559 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.559 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.559 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.559 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.559 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.560 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.560 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.560 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iostat", 2024-03-26T06:47:37.560 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.560 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.560 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.560 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.561 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.561 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.561 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.561 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.561 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.561 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.561 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.561 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.561 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.562 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.562 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.562 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.562 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.562 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.562 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.562 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.562 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.563 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.564 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.564 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.564 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.564 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.564 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.564 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.564 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.564 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.565 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.565 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.565 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.565 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.565 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.565 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.565 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.565 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.565 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.566 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.566 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.566 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.566 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.566 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.566 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.566 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.566 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.567 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.567 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.567 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.567 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.567 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.567 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.567 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.567 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.568 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.568 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.568 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.568 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.568 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.568 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.568 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.568 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.569 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.569 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.569 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.569 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "k8sevents", 2024-03-26T06:47:37.569 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.569 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.569 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.570 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_event_retention_days": { 2024-03-26T06:47:37.570 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ceph_event_retention_days", 2024-03-26T06:47:37.570 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.570 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.570 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.570 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7", 2024-03-26T06:47:37.570 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.570 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.570 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.571 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-26T06:47:37.571 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.571 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.571 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.571 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.571 INFO:teuthology.orchestra.run.smithi038.stdout: "config_check_secs": { 2024-03-26T06:47:37.571 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_check_secs", 2024-03-26T06:47:37.571 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.572 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.572 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.572 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:37.572 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "10", 2024-03-26T06:47:37.572 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.572 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.572 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-26T06:47:37.572 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.573 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.573 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.573 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.573 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.573 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.573 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.573 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.573 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.573 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.574 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.574 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.574 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.574 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.574 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.574 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.574 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.574 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.575 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.575 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.575 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.575 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.575 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.575 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.575 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.576 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.576 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.576 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.576 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.576 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.576 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.576 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.576 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.577 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.577 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.577 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.577 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.577 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.577 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.577 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.577 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.577 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.578 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.578 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.578 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.578 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.578 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.578 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.578 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.578 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.579 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.579 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.579 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.579 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.579 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.579 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.579 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.579 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.580 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.580 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.580 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.580 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.580 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.580 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.580 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.580 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.581 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.581 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.581 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.581 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.581 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.581 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.581 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.581 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.582 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.582 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "localpool", 2024-03-26T06:47:37.582 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.582 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.582 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.582 INFO:teuthology.orchestra.run.smithi038.stdout: "failure_domain": { 2024-03-26T06:47:37.582 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "failure_domain", 2024-03-26T06:47:37.582 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.583 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.583 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.583 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "host", 2024-03-26T06:47:37.583 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.583 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.583 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.583 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "failure domain for any created local pool", 2024-03-26T06:47:37.583 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-26T06:47:37.583 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.584 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.584 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.584 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.584 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.584 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.584 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.584 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.585 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.585 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.585 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.585 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.585 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.585 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.585 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.585 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.585 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.586 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.586 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.586 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.586 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.586 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.586 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.586 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.590 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.590 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.590 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.590 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.590 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.590 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.591 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.591 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.591 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.591 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.591 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.591 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.591 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.592 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.592 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.592 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.592 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.592 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.592 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.592 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.592 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.592 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.593 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.593 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.593 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.593 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.593 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.593 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.593 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.593 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.594 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.594 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.594 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.594 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.594 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.594 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.594 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.594 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.594 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.595 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.595 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.595 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.595 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.595 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.595 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.595 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": { 2024-03-26T06:47:37.596 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_size", 2024-03-26T06:47:37.596 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.596 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.596 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.596 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.596 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.596 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.596 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.596 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default min_size for any created local pool", 2024-03-26T06:47:37.597 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-26T06:47:37.597 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.597 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.597 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.597 INFO:teuthology.orchestra.run.smithi038.stdout: "num_rep": { 2024-03-26T06:47:37.597 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "num_rep", 2024-03-26T06:47:37.597 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.597 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.598 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.598 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3", 2024-03-26T06:47:37.598 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.598 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.598 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.598 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default replica count for any created local pool", 2024-03-26T06:47:37.598 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.599 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.599 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.599 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.599 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": { 2024-03-26T06:47:37.599 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_num", 2024-03-26T06:47:37.599 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.599 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.599 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.600 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "128", 2024-03-26T06:47:37.600 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.600 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.600 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.600 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default pg_num for any created local pool", 2024-03-26T06:47:37.600 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.600 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.600 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.601 INFO:teuthology.orchestra.run.smithi038.stdout: "prefix": { 2024-03-26T06:47:37.601 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prefix", 2024-03-26T06:47:37.601 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.601 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.601 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.601 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.601 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.601 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.602 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.602 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name prefix for any created local pool", 2024-03-26T06:47:37.602 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.602 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.602 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.602 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.602 INFO:teuthology.orchestra.run.smithi038.stdout: "subtree": { 2024-03-26T06:47:37.602 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "subtree", 2024-03-26T06:47:37.603 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.603 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.603 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.603 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "rack", 2024-03-26T06:47:37.603 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.603 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.603 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.603 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-26T06:47:37.603 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-26T06:47:37.604 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.604 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.604 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.604 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.604 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.604 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.604 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mds_autoscaler", 2024-03-26T06:47:37.604 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.605 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.605 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.605 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.605 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.605 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.605 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.605 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.605 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.605 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.606 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.606 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.606 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.606 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.606 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.606 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.606 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.606 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.606 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.607 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.607 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.607 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.607 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.607 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.607 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.607 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.607 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.608 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.608 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.608 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.608 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.608 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.608 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.608 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.608 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.609 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.609 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.609 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.609 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.609 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.609 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.609 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.609 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.610 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.610 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.610 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.610 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.610 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.610 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.610 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.610 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.611 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.611 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.611 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.611 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.611 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.611 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.611 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.611 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.612 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.612 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.612 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.612 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.612 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.612 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.612 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.612 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.613 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.613 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.613 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.613 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.613 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirroring", 2024-03-26T06:47:37.613 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.614 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.614 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.614 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.614 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.614 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.614 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.614 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.614 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.615 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.615 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.615 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.615 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.615 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.615 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.615 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.615 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.615 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.616 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.616 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.616 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.616 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.616 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.616 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.616 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.616 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.617 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.617 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.617 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.617 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.617 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.617 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.617 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.617 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.617 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.618 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.618 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.618 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.618 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.618 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.618 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.619 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.619 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.619 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.619 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.619 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.619 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.619 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.619 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.619 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.620 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.620 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.620 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.620 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.620 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.620 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.620 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.620 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.621 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.621 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.621 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.621 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.621 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.621 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.621 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.621 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.622 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.622 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.622 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.622 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.622 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.622 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.622 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.623 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "nfs", 2024-03-26T06:47:37.623 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.623 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.623 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.623 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.623 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.623 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.623 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.624 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.624 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.624 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.624 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.624 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.624 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.624 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.624 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.625 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.625 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.625 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.625 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.625 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.625 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.625 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.625 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.626 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.626 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.626 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.626 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.626 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.626 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.626 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.626 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.627 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.627 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.627 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.627 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.627 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.627 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.627 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.628 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.628 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.628 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.628 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.628 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.628 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.628 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.628 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.628 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.629 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.629 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.629 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.629 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.629 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.629 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.629 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.629 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.630 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.630 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.630 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.630 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.630 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.630 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.630 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.630 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.631 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.631 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.631 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.631 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.631 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.631 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.632 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-03-26T06:47:37.632 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.632 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.632 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.632 INFO:teuthology.orchestra.run.smithi038.stdout: "fail_fs": { 2024-03-26T06:47:37.632 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "fail_fs", 2024-03-26T06:47:37.632 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.633 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.633 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.633 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.633 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.633 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.633 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.633 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-26T06:47:37.633 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.634 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.634 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.634 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.634 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.634 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.634 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.634 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.635 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.635 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.635 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.635 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.635 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.635 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.635 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.635 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.635 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.636 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.636 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.636 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.636 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.636 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.636 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.636 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.637 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.637 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.637 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.637 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.637 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.637 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.637 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.637 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.638 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.638 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.638 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.638 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.638 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.638 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.638 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.638 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.638 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.639 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.639 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.639 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.639 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.639 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.639 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.639 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.639 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.640 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.640 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.640 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.640 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.640 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.640 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.640 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.640 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.641 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.641 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.641 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.641 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.641 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.641 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.641 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.641 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.642 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.642 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.642 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.642 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.642 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.642 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator": { 2024-03-26T06:47:37.642 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-03-26T06:47:37.642 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.643 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.643 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.643 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.643 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.643 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.643 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.643 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-03-26T06:47:37.643 INFO:teuthology.orchestra.run.smithi038.stdout: "rook", 2024-03-26T06:47:37.644 INFO:teuthology.orchestra.run.smithi038.stdout: "test_orchestrator" 2024-03-26T06:47:37.644 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.644 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Orchestrator backend", 2024-03-26T06:47:37.644 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.644 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.644 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.644 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.644 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.645 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.645 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_perf_query", 2024-03-26T06:47:37.645 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.645 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.645 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.645 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.645 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.645 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.646 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.647 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.648 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.649 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.650 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.651 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.652 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.653 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_support", 2024-03-26T06:47:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.654 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.655 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.656 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.656 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.656 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.656 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.656 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.656 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.656 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.656 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.657 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.657 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.657 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.657 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.657 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.657 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.657 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.658 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.658 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.658 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.658 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.658 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.658 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.658 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.658 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.659 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.659 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.659 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.659 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.659 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.659 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.659 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.659 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.660 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.660 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.660 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.660 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.660 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.660 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.660 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.661 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.661 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.661 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.661 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.661 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.661 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.661 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.662 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.662 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.662 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.662 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.662 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.662 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.662 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.662 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.663 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.663 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.663 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.663 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.663 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.663 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.663 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_autoscaler", 2024-03-26T06:47:37.664 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.664 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.664 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.664 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.664 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.664 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.664 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.664 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.664 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.665 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.665 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.665 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.665 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.665 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.665 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.665 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.665 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.666 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.666 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.666 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.666 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.666 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.666 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.666 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.666 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.667 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.667 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.667 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.667 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.667 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.667 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.667 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.667 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.668 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.668 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.668 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.668 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.668 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.668 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.668 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.668 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.668 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.669 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.669 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.669 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.669 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.669 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.669 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.669 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.669 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.670 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.670 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.670 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.670 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.670 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.670 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.670 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.670 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.671 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.671 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.671 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.671 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.671 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.671 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.671 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.671 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.671 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.672 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.672 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.672 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.672 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.672 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.672 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.672 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:37.672 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:37.673 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.673 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.673 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.673 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:37.673 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.673 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.673 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.673 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.674 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.674 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.674 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.674 INFO:teuthology.orchestra.run.smithi038.stdout: "threshold": { 2024-03-26T06:47:37.674 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threshold", 2024-03-26T06:47:37.674 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:37.674 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.674 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.675 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-03-26T06:47:37.675 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1.0", 2024-03-26T06:47:37.675 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.675 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.675 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "scaling threshold", 2024-03-26T06:47:37.675 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-26T06:47:37.675 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.675 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.676 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.676 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.676 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.676 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "progress", 2024-03-26T06:47:37.676 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.676 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.676 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.676 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_pg_recovery_event": { 2024-03-26T06:47:37.677 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_pg_recovery_event", 2024-03-26T06:47:37.677 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.677 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.677 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.677 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.677 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.677 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.677 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.678 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-26T06:47:37.678 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.678 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.678 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.678 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-03-26T06:47:37.678 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-03-26T06:47:37.678 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.679 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.679 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.679 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.679 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.679 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.679 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.679 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.679 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.680 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.680 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.680 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.680 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.680 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.680 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.680 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.681 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.681 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.681 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.681 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.681 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.681 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.681 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.681 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.682 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.682 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.682 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.682 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.682 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.682 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.682 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.683 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.684 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.685 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.685 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.685 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.685 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.685 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.685 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.685 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.685 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.686 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.686 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.686 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.686 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.686 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.686 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.686 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.686 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.687 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.687 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.687 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.687 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.687 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.687 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.687 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.687 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.688 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.688 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.688 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.688 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.688 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.688 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.688 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.689 INFO:teuthology.orchestra.run.smithi038.stdout: "max_completed_events": { 2024-03-26T06:47:37.689 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_completed_events", 2024-03-26T06:47:37.689 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.689 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.689 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.689 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "50", 2024-03-26T06:47:37.689 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.689 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.689 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.690 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "number of past completed events to remember", 2024-03-26T06:47:37.690 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.690 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.690 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.690 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:37.690 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:37.690 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.691 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.691 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.691 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:37.691 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.691 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.691 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.691 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long the module is going to sleep", 2024-03-26T06:47:37.691 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.692 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.692 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.692 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.692 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.692 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus", 2024-03-26T06:47:37.692 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.693 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.693 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.693 INFO:teuthology.orchestra.run.smithi038.stdout: "cache": { 2024-03-26T06:47:37.693 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cache", 2024-03-26T06:47:37.693 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.693 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.693 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.693 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.693 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.694 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.694 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.694 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.694 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.694 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.694 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.694 INFO:teuthology.orchestra.run.smithi038.stdout: "exclude_perf_counters": { 2024-03-26T06:47:37.695 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "exclude_perf_counters", 2024-03-26T06:47:37.695 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.695 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.695 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.695 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.695 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.695 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.695 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.696 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-26T06:47:37.696 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-26T06:47:37.696 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.696 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.696 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.696 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.696 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.697 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.697 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.697 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.697 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.697 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.697 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.697 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.698 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.698 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.698 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.698 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.698 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.698 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.698 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.698 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.699 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.700 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.701 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.702 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.703 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.704 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools": { 2024-03-26T06:47:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools", 2024-03-26T06:47:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.705 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-26T06:47:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-26T06:47:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "300", 2024-03-26T06:47:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.707 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_interval": { 2024-03-26T06:47:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_interval", 2024-03-26T06:47:37.708 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15.0", 2024-03-26T06:47:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.709 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-03-26T06:47:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-03-26T06:47:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.710 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-03-26T06:47:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-26T06:47:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.711 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-03-26T06:47:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-03-26T06:47:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "9283", 2024-03-26T06:47:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.712 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-26T06:47:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: "stale_cache_strategy": { 2024-03-26T06:47:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stale_cache_strategy", 2024-03-26T06:47:37.713 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.714 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.714 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.714 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "log", 2024-03-26T06:47:37.714 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.714 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.714 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.714 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.714 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.715 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.715 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.715 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.715 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-03-26T06:47:37.715 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-03-26T06:47:37.715 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.715 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.715 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "default", 2024-03-26T06:47:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "default", 2024-03-26T06:47:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "error" 2024-03-26T06:47:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.716 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-03-26T06:47:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-03-26T06:47:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.717 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-03-26T06:47:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-03-26T06:47:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-03-26T06:47:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.718 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-03-26T06:47:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.719 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.720 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.721 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.722 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.723 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.724 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.725 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.726 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.727 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.727 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.727 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.727 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.727 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.727 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.727 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.728 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.728 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.728 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.728 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.728 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.728 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.728 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_snap_create": { 2024-03-26T06:47:37.728 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_snap_create", 2024-03-26T06:47:37.729 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.729 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.729 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.729 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:37.729 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.729 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.729 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.729 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.729 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.730 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.730 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.730 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.730 INFO:teuthology.orchestra.run.smithi038.stdout: "mirror_snapshot_schedule": { 2024-03-26T06:47:37.730 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirror_snapshot_schedule", 2024-03-26T06:47:37.730 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.730 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.730 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.731 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.731 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.731 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.731 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.731 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.731 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.731 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.731 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.732 INFO:teuthology.orchestra.run.smithi038.stdout: "trash_purge_schedule": { 2024-03-26T06:47:37.732 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "trash_purge_schedule", 2024-03-26T06:47:37.732 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.732 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.732 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.732 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.732 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.733 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.733 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.733 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.733 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.733 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.733 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.733 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.733 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.734 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.734 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "restful", 2024-03-26T06:47:37.734 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.734 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.734 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.734 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_auth": { 2024-03-26T06:47:37.734 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_auth", 2024-03-26T06:47:37.735 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.735 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.735 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.735 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.735 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.735 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.735 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.735 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.735 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.736 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.736 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.736 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-03-26T06:47:37.736 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-03-26T06:47:37.736 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.736 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.736 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.737 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.737 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.737 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.737 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.737 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.737 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.737 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.737 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.738 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.738 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.738 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.738 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.738 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.739 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.739 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.739 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.739 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.739 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.739 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.739 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.739 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.740 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.740 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.740 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.740 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.740 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.740 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.740 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.740 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.741 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.741 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.741 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.741 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.741 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.741 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.741 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.741 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.742 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.742 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.742 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.742 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.742 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.742 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.742 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.742 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.743 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.743 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.743 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.743 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.743 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.743 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.743 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.743 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.743 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.744 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.744 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.744 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.744 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.744 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.744 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.744 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.744 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.744 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.745 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.745 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.745 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.745 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.745 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.745 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.745 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.745 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.746 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.746 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.746 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.746 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.746 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.746 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.746 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-03-26T06:47:37.747 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-03-26T06:47:37.747 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.747 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.747 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.747 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.747 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.747 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.747 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.748 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.748 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.748 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.748 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.748 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-03-26T06:47:37.748 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-03-26T06:47:37.748 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.749 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.749 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.749 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.749 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.749 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.749 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.749 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.749 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.750 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.750 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.750 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.750 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.750 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.750 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.750 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw", 2024-03-26T06:47:37.750 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.751 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.751 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.751 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.751 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.751 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.751 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.751 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.751 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.751 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.752 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.752 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.752 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.752 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.752 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.752 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.752 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.752 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.753 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.753 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.753 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.753 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.753 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.753 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.753 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.754 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.754 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.754 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.754 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.754 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.754 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.754 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.754 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.755 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.755 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.755 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.755 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.755 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.755 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.755 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.755 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.756 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.756 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.756 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.756 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.756 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.756 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.756 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.756 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.757 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.757 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.757 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.757 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.757 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.757 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.757 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.757 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.758 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.758 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.758 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.758 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.758 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.758 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.758 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.759 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.759 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.759 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.759 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.759 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.759 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.759 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.760 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.760 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rook", 2024-03-26T06:47:37.760 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.760 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.760 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.760 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.760 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.760 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.761 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.761 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.761 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.761 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.761 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.761 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.761 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.761 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.762 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.762 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.762 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.762 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.762 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.762 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.762 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.762 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.763 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.763 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.763 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.763 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.763 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.763 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.763 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.763 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.764 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.764 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.764 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.764 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.764 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.764 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.764 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.765 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.765 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.765 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.765 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.765 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.765 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.765 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.765 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.766 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.766 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.766 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.766 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.766 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.766 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.766 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.766 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.767 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.767 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.767 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.767 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.767 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.767 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.767 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.767 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.768 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.768 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.768 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.768 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.768 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.768 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.768 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.768 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.769 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.769 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.769 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_class": { 2024-03-26T06:47:37.769 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "storage_class", 2024-03-26T06:47:37.769 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.769 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.769 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.769 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "local", 2024-03-26T06:47:37.769 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.770 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.770 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.770 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-26T06:47:37.770 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.770 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.770 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.771 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.771 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "selftest", 2024-03-26T06:47:37.771 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.771 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.771 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.771 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.771 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.772 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.772 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.772 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.772 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.772 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.772 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.772 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.772 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.772 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.773 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.773 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.773 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.773 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.773 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.773 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.773 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.773 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.774 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.774 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.774 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.774 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.774 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.774 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.774 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.774 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.774 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.775 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.775 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.775 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.775 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.775 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.775 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.775 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.776 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.776 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.776 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.776 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.776 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.776 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.776 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.776 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.777 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.777 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.777 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.777 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.777 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.777 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.777 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.777 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.778 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.779 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "roption1": { 2024-03-26T06:47:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption1", 2024-03-26T06:47:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.780 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.781 INFO:teuthology.orchestra.run.smithi038.stdout: "roption2": { 2024-03-26T06:47:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption2", 2024-03-26T06:47:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "xyz", 2024-03-26T06:47:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.782 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption1": { 2024-03-26T06:47:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption1", 2024-03-26T06:47:37.783 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.784 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption2": { 2024-03-26T06:47:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption2", 2024-03-26T06:47:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.785 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.786 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption3": { 2024-03-26T06:47:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption3", 2024-03-26T06:47:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.787 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption4": { 2024-03-26T06:47:37.788 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption4", 2024-03-26T06:47:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.789 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.790 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption5": { 2024-03-26T06:47:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption5", 2024-03-26T06:47:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.793 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.794 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption6": { 2024-03-26T06:47:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption6", 2024-03-26T06:47:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.795 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption7": { 2024-03-26T06:47:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption7", 2024-03-26T06:47:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "42", 2024-03-26T06:47:37.797 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: "testkey": { 2024-03-26T06:47:37.798 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testkey", 2024-03-26T06:47:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.799 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "testlkey": { 2024-03-26T06:47:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testlkey", 2024-03-26T06:47:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.800 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.801 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "testnewline": { 2024-03-26T06:47:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testnewline", 2024-03-26T06:47:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.802 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.803 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-03-26T06:47:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_m_granularity": { 2024-03-26T06:47:37.804 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_m_granularity", 2024-03-26T06:47:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.805 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow minute scheduled snapshots", 2024-03-26T06:47:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "dump_on_update": { 2024-03-26T06:47:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dump_on_update", 2024-03-26T06:47:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.806 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "dump database to debug log on update", 2024-03-26T06:47:37.807 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.808 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.809 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.810 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.811 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.812 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.812 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.812 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.812 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.812 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.812 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.812 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.813 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.814 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.815 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.816 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stats", 2024-03-26T06:47:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.817 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.818 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.819 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.820 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.821 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.822 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.823 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.824 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.825 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.826 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.826 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.826 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "status", 2024-03-26T06:47:37.826 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.826 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.826 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.827 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.827 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.827 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.827 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.827 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.827 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.827 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.827 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.828 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.828 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.828 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.828 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.828 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.828 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.828 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.828 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.828 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.829 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.829 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.829 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.829 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.829 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.829 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.829 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.829 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.830 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.830 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.830 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.830 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.830 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.830 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.830 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.830 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.830 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.831 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.831 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.831 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.831 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.831 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.831 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.831 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.832 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.832 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.832 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.832 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.832 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.832 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.832 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.832 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.833 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.833 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.833 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.833 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.833 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.833 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.833 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.834 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.834 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.834 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.834 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.834 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.834 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.834 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.834 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.834 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.835 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.835 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.835 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.835 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.835 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.835 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telegraf", 2024-03-26T06:47:37.835 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.836 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.836 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.836 INFO:teuthology.orchestra.run.smithi038.stdout: "address": { 2024-03-26T06:47:37.836 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "address", 2024-03-26T06:47:37.836 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.836 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.836 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-26T06:47:37.837 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.837 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.837 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.837 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.837 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.837 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.837 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.838 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.838 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:37.838 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:37.838 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.838 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.838 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.838 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15", 2024-03-26T06:47:37.838 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.839 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.839 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.839 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.839 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.839 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.839 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.839 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.840 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.840 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.840 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.840 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.840 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.840 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.840 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.840 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.841 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.841 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.841 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.841 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.841 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.841 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.841 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.841 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.842 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.842 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.842 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.842 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.842 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.842 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.842 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.842 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.843 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.843 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.843 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.843 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.843 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.843 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.843 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.843 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.844 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.844 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.844 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.844 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.844 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.844 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.844 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.844 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.845 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.845 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.845 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.845 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.845 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.845 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.845 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.845 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.846 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.846 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.846 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.846 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.846 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.846 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.846 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.847 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.847 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.847 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.847 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.847 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.847 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.847 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.847 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.848 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.848 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.848 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.848 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.848 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.848 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.848 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.849 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telemetry", 2024-03-26T06:47:37.849 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.849 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.849 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.849 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_basic": { 2024-03-26T06:47:37.849 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_basic", 2024-03-26T06:47:37.849 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.849 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.850 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.850 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.850 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.850 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.850 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.850 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-26T06:47:37.850 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.850 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.851 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.851 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_crash": { 2024-03-26T06:47:37.851 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_crash", 2024-03-26T06:47:37.851 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.851 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.851 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.851 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.851 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.852 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.852 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.852 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-26T06:47:37.852 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.852 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.852 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.852 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.852 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_device": { 2024-03-26T06:47:37.853 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_device", 2024-03-26T06:47:37.853 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.853 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.853 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.853 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.853 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.853 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.853 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.853 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-26T06:47:37.854 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.854 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.854 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.854 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.854 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_ident": { 2024-03-26T06:47:37.854 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_ident", 2024-03-26T06:47:37.854 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.854 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.855 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.855 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.855 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.855 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.855 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.855 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-26T06:47:37.855 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.855 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.856 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.856 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_perf": { 2024-03-26T06:47:37.856 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_perf", 2024-03-26T06:47:37.856 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.856 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.856 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.856 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-26T06:47:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.857 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "contact": { 2024-03-26T06:47:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "contact", 2024-03-26T06:47:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.858 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: "description": { 2024-03-26T06:47:37.859 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "description", 2024-03-26T06:47:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.860 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: "device_url": { 2024-03-26T06:47:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_url", 2024-03-26T06:47:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.861 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-26T06:47:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.862 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-03-26T06:47:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-03-26T06:47:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.863 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.864 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "24", 2024-03-26T06:47:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "8", 2024-03-26T06:47:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.865 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.866 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.866 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.866 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.866 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.866 INFO:teuthology.orchestra.run.smithi038.stdout: "last_opt_revision": { 2024-03-26T06:47:37.866 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "last_opt_revision", 2024-03-26T06:47:37.866 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.867 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.867 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.867 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1", 2024-03-26T06:47:37.867 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.867 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.867 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.867 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.867 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.868 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.868 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.868 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard": { 2024-03-26T06:47:37.868 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard", 2024-03-26T06:47:37.868 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.868 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.868 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.868 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.869 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.869 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.869 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.869 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.869 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.869 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.869 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.869 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.870 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard_description": { 2024-03-26T06:47:37.870 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard_description", 2024-03-26T06:47:37.870 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.870 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.870 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.870 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.870 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.870 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.871 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.871 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.871 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.871 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.871 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.871 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.871 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.872 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.872 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.872 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.872 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.872 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.872 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.872 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.872 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.872 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.873 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.873 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.873 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.873 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.873 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.873 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.873 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.873 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.874 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.874 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.874 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.874 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.874 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.874 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.874 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.874 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.874 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.875 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.875 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.875 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.875 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.875 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.875 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.875 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.876 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.876 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.876 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.876 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.876 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.876 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.876 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.876 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.877 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.877 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.877 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.877 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.877 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.877 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.877 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.877 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.878 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.879 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.879 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.879 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.879 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.879 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.879 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.879 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.879 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.880 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.880 INFO:teuthology.orchestra.run.smithi038.stdout: "organization": { 2024-03-26T06:47:37.880 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "organization", 2024-03-26T06:47:37.880 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.880 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.880 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.880 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.881 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.881 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.881 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.881 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.881 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.881 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.881 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.881 INFO:teuthology.orchestra.run.smithi038.stdout: "proxy": { 2024-03-26T06:47:37.882 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "proxy", 2024-03-26T06:47:37.882 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.882 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.882 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.882 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.882 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.882 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.882 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.883 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.883 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.883 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.883 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.883 INFO:teuthology.orchestra.run.smithi038.stdout: "url": { 2024-03-26T06:47:37.883 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url", 2024-03-26T06:47:37.883 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.884 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.884 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.884 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-26T06:47:37.884 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.884 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.884 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.884 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.884 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.885 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.885 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.885 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.885 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.885 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.885 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test_orchestrator", 2024-03-26T06:47:37.885 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.886 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.886 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.886 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.886 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.886 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.886 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.886 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.886 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.887 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.887 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.887 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.887 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.887 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.887 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.887 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.887 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.888 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.888 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.888 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.888 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.888 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.888 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.888 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.889 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.889 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.889 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.889 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.889 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.889 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.889 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.889 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.889 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.890 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.890 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.890 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.890 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.890 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.890 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.890 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.891 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.891 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.891 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.891 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.891 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.891 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.891 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.891 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.891 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.892 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.892 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.892 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.892 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.892 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.892 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.892 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.892 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.892 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.893 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.893 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.893 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.893 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.893 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.893 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.893 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.893 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.894 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.894 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.894 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.894 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.894 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.894 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.894 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.895 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-03-26T06:47:37.895 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.895 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.895 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.895 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.895 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.895 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.895 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.895 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.896 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.896 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.896 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.896 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.896 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.896 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.896 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.896 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.897 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.897 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.897 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.897 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.897 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.897 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.897 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.897 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.898 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.898 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.898 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.898 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.898 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.898 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.898 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.898 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.899 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.899 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.899 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.899 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.899 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.899 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.899 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.899 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.900 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.900 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.900 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.900 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.900 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.900 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.900 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.900 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.901 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.901 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.901 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.901 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.901 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.901 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.901 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.901 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.902 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.902 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.902 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.902 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.902 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.902 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.902 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.902 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.902 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.903 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.903 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.903 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.903 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.903 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.903 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.903 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.904 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_clones": { 2024-03-26T06:47:37.904 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_clones", 2024-03-26T06:47:37.904 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.904 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.904 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.904 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4", 2024-03-26T06:47:37.904 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.904 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.905 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.905 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-26T06:47:37.905 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.905 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.905 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.905 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.905 INFO:teuthology.orchestra.run.smithi038.stdout: "periodic_async_work": { 2024-03-26T06:47:37.905 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "periodic_async_work", 2024-03-26T06:47:37.905 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.906 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.906 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.906 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.906 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.906 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.906 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.906 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Periodically check for async work", 2024-03-26T06:47:37.906 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.907 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.907 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.907 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.907 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_delay": { 2024-03-26T06:47:37.907 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_delay", 2024-03-26T06:47:37.907 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.907 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.908 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.908 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:37.908 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.908 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.908 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.908 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-26T06:47:37.908 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.908 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.908 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.909 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.909 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_no_wait": { 2024-03-26T06:47:37.909 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_no_wait", 2024-03-26T06:47:37.909 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.909 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.909 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.909 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.909 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.910 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.910 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.910 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-26T06:47:37.910 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.910 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.910 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.910 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.911 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.911 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.911 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix", 2024-03-26T06:47:37.911 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.911 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.911 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.911 INFO:teuthology.orchestra.run.smithi038.stdout: "discovery_interval": { 2024-03-26T06:47:37.912 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "discovery_interval", 2024-03-26T06:47:37.912 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:37.912 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.912 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.912 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "100", 2024-03-26T06:47:37.912 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.912 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.912 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.912 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.913 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.913 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.913 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.913 INFO:teuthology.orchestra.run.smithi038.stdout: "identifier": { 2024-03-26T06:47:37.913 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "identifier", 2024-03-26T06:47:37.913 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.913 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.914 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.914 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.914 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.914 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.914 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.914 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.914 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.914 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.915 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.915 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:37.915 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:37.915 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.915 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.915 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.915 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:37.915 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.916 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.916 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.916 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.916 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.916 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.916 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.916 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.916 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.917 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.917 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.917 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.917 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.917 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.917 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.917 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.917 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.918 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.918 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.918 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.918 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.918 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.918 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.918 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.918 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.919 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.919 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.919 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.919 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.919 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.919 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.919 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.919 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.920 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.920 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.920 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.920 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.920 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.920 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.920 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.920 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.920 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.921 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.921 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.921 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.921 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.921 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.921 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.921 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.921 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.922 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.922 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.922 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.922 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.922 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.922 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.922 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.922 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.923 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.923 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.923 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.923 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.923 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.923 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.923 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.924 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.924 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.924 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.924 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.924 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.924 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.924 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.924 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.925 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.925 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.925 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.925 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_host": { 2024-03-26T06:47:37.925 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_host", 2024-03-26T06:47:37.925 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.925 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.926 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.926 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.926 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.926 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.926 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.926 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.926 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.926 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.927 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.927 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_port": { 2024-03-26T06:47:37.927 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_port", 2024-03-26T06:47:37.927 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.927 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.927 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.927 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10051", 2024-03-26T06:47:37.928 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.928 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.928 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.928 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.928 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.928 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.928 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.928 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.928 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_sender": { 2024-03-26T06:47:37.929 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_sender", 2024-03-26T06:47:37.929 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.929 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.929 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.929 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-26T06:47:37.929 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.929 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.929 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.930 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.930 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.930 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.930 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.930 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.930 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:47:37.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.931 INFO:teuthology.orchestra.run.smithi038.stdout: "gid": 24962, 2024-03-26T06:47:37.931 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "y", 2024-03-26T06:47:37.931 INFO:teuthology.orchestra.run.smithi038.stdout: "mgr_features": 4540701547738038271, 2024-03-26T06:47:37.931 INFO:teuthology.orchestra.run.smithi038.stdout: "available_modules": [ 2024-03-26T06:47:37.931 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.931 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "alerts", 2024-03-26T06:47:37.932 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.932 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.932 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.932 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:37.932 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:37.932 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.932 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.932 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.932 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:37.933 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.933 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.933 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.933 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How frequently to reexamine health status", 2024-03-26T06:47:37.933 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.933 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.933 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.934 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.934 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.934 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.934 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.934 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.934 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.934 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.934 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.934 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.935 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.935 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.935 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.935 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.935 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.935 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.935 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.935 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.936 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.936 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.936 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.936 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.936 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.936 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.936 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.937 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.937 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.937 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.937 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.937 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.937 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.937 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.938 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.938 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.938 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.938 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.938 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.938 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.938 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.939 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.939 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.939 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.939 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.939 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.939 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.939 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.939 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.939 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.940 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.940 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.940 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.940 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.940 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.940 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.940 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.941 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.941 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.941 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.941 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.941 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.941 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.941 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.941 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.942 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.942 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.942 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.942 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.942 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.942 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_destination": { 2024-03-26T06:47:37.942 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_destination", 2024-03-26T06:47:37.942 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.942 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.943 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.943 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.943 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.943 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.943 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.943 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email address to send alerts to", 2024-03-26T06:47:37.943 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.943 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.944 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.944 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.944 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_from_name": { 2024-03-26T06:47:37.944 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_from_name", 2024-03-26T06:47:37.944 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.944 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.944 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.944 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "Ceph", 2024-03-26T06:47:37.945 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.945 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.945 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.945 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email From: name", 2024-03-26T06:47:37.945 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.945 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.945 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.945 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_host": { 2024-03-26T06:47:37.946 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_host", 2024-03-26T06:47:37.946 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.946 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.946 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.946 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.946 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.946 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.946 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.947 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP server", 2024-03-26T06:47:37.947 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.947 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.947 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.947 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_password": { 2024-03-26T06:47:37.947 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_password", 2024-03-26T06:47:37.947 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.948 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.948 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.948 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.948 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.948 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.948 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.948 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Password to authenticate with", 2024-03-26T06:47:37.948 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.949 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.949 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.949 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_port": { 2024-03-26T06:47:37.949 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_port", 2024-03-26T06:47:37.949 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.949 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.949 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.950 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "465", 2024-03-26T06:47:37.950 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.950 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.950 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.950 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP port", 2024-03-26T06:47:37.950 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.950 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.950 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.951 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_sender": { 2024-03-26T06:47:37.951 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_sender", 2024-03-26T06:47:37.951 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.951 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.951 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.951 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.951 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.951 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.952 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.952 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP envelope sender", 2024-03-26T06:47:37.952 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.952 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.952 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.952 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.952 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_ssl": { 2024-03-26T06:47:37.952 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_ssl", 2024-03-26T06:47:37.953 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.953 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.953 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.953 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.953 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.953 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.953 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.953 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-26T06:47:37.953 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.954 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.954 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.954 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.954 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_user": { 2024-03-26T06:47:37.954 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_user", 2024-03-26T06:47:37.954 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.954 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.955 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.955 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.955 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.955 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.955 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.955 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "User to authenticate as", 2024-03-26T06:47:37.955 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.955 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.955 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.956 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.956 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.956 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.956 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.956 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "balancer", 2024-03-26T06:47:37.956 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.956 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.956 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.956 INFO:teuthology.orchestra.run.smithi038.stdout: "active": { 2024-03-26T06:47:37.957 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "active", 2024-03-26T06:47:37.957 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.957 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.957 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.957 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:37.957 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.957 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.957 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.958 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically balance PGs across cluster", 2024-03-26T06:47:37.958 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.958 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.958 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.958 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_time": { 2024-03-26T06:47:37.958 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_time", 2024-03-26T06:47:37.958 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.959 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.959 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.959 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0000", 2024-03-26T06:47:37.959 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.959 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.959 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.959 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "beginning time of day to automatically balance", 2024-03-26T06:47:37.959 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-26T06:47:37.960 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.960 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.960 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.960 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_weekday": { 2024-03-26T06:47:37.960 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_weekday", 2024-03-26T06:47:37.960 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:37.960 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.960 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.961 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:37.961 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-03-26T06:47:37.961 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-03-26T06:47:37.961 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.961 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-26T06:47:37.961 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-26T06:47:37.961 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.961 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.962 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_max_iterations": { 2024-03-26T06:47:37.962 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_max_iterations", 2024-03-26T06:47:37.962 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:37.962 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.962 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.962 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "25", 2024-03-26T06:47:37.962 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:37.962 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "250", 2024-03-26T06:47:37.963 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.963 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-26T06:47:37.963 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.963 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.963 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.963 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_metrics": { 2024-03-26T06:47:37.963 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_metrics", 2024-03-26T06:47:37.963 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.964 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.964 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.964 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "pgs,objects,bytes", 2024-03-26T06:47:37.964 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.964 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.964 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.964 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-26T06:47:37.964 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-26T06:47:37.965 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.965 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.965 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.965 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_step": { 2024-03-26T06:47:37.965 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_step", 2024-03-26T06:47:37.965 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:37.965 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.965 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.965 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.5", 2024-03-26T06:47:37.966 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0.001", 2024-03-26T06:47:37.966 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "0.999", 2024-03-26T06:47:37.966 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.966 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "aggressiveness of optimization", 2024-03-26T06:47:37.966 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-26T06:47:37.966 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.966 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.967 INFO:teuthology.orchestra.run.smithi038.stdout: "end_time": { 2024-03-26T06:47:37.967 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_time", 2024-03-26T06:47:37.967 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.967 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.967 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.967 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2359", 2024-03-26T06:47:37.967 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.967 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.968 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.968 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ending time of day to automatically balance", 2024-03-26T06:47:37.968 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-26T06:47:37.968 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.968 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.968 INFO:teuthology.orchestra.run.smithi038.stdout: "end_weekday": { 2024-03-26T06:47:37.968 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_weekday", 2024-03-26T06:47:37.968 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:37.969 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.969 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.969 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:37.969 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-03-26T06:47:37.969 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-03-26T06:47:37.969 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.969 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-26T06:47:37.969 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-26T06:47:37.970 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.970 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.970 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.970 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:37.970 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:37.970 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.970 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.970 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.971 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.971 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.971 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.971 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.971 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.971 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.971 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.971 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.972 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.972 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.972 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.972 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.972 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.972 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.972 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.972 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.972 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:37.973 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:37.973 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.973 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.973 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.973 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.973 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.973 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.973 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.973 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.974 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.974 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.974 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.974 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:37.974 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:37.974 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.974 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.975 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.975 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:37.975 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.975 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.975 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.975 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:37.975 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:37.975 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:37.976 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:37.976 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:37.976 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:37.976 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.976 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.976 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.976 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.976 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.977 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.977 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:37.977 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:37.977 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.977 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.977 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.977 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.977 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.978 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.978 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.978 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:37.978 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.978 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.978 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.978 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.978 INFO:teuthology.orchestra.run.smithi038.stdout: "min_score": { 2024-03-26T06:47:37.979 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_score", 2024-03-26T06:47:37.979 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:37.979 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.979 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.979 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:37.979 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.979 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.979 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.980 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-26T06:47:37.980 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.980 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.980 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.980 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-03-26T06:47:37.980 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-03-26T06:47:37.980 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.981 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.981 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.981 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "upmap", 2024-03-26T06:47:37.981 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.981 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.981 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:37.981 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-compat", 2024-03-26T06:47:37.981 INFO:teuthology.orchestra.run.smithi038.stdout: "none", 2024-03-26T06:47:37.981 INFO:teuthology.orchestra.run.smithi038.stdout: "read", 2024-03-26T06:47:37.982 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap", 2024-03-26T06:47:37.982 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap-read" 2024-03-26T06:47:37.982 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:37.982 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Balancer mode", 2024-03-26T06:47:37.982 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.982 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.982 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.983 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_ids": { 2024-03-26T06:47:37.983 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_ids", 2024-03-26T06:47:37.983 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:37.983 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.983 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.983 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:37.983 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.983 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.983 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.984 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-26T06:47:37.984 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.984 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.984 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.984 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:37.984 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:37.984 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.985 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.985 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.985 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:37.985 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.985 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.985 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.985 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-26T06:47:37.985 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.986 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.986 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.986 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_deviation": { 2024-03-26T06:47:37.986 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_deviation", 2024-03-26T06:47:37.986 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.986 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.986 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.987 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:37.987 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:37.987 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.987 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.987 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-26T06:47:37.987 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-26T06:47:37.987 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.987 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.988 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_optimizations": { 2024-03-26T06:47:37.988 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_optimizations", 2024-03-26T06:47:37.988 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:37.988 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.988 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:37.988 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:37.988 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.988 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.989 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.989 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-26T06:47:37.989 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.989 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.989 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.989 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:37.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.989 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:37.990 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm", 2024-03-26T06:47:37.990 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:37.990 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:37.990 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:37.990 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_down_multiplier": { 2024-03-26T06:47:37.990 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_down_multiplier", 2024-03-26T06:47:37.990 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:37.990 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.991 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.991 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-03-26T06:47:37.991 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.991 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.991 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.991 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-26T06:47:37.991 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.991 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.991 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.992 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_refresh_rate": { 2024-03-26T06:47:37.992 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_refresh_rate", 2024-03-26T06:47:37.992 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.992 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.992 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.992 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "20", 2024-03-26T06:47:37.992 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.993 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.993 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.993 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-26T06:47:37.993 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.993 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.993 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.993 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_starting_port": { 2024-03-26T06:47:37.994 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_starting_port", 2024-03-26T06:47:37.994 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:37.994 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.994 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.994 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4721", 2024-03-26T06:47:37.994 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.994 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.994 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.994 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-26T06:47:37.995 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.995 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.995 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.995 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_ptrace": { 2024-03-26T06:47:37.995 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_ptrace", 2024-03-26T06:47:37.995 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:37.995 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.996 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.996 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:37.996 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.996 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.996 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.996 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-26T06:47:37.996 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-26T06:47:37.996 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.997 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.997 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_interval": { 2024-03-26T06:47:37.997 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_interval", 2024-03-26T06:47:37.997 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:37.997 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.997 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.997 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:37.998 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.998 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:37.998 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:37.998 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-26T06:47:37.998 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:37.998 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:37.998 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:37.998 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:37.999 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_memory_target_ratio": { 2024-03-26T06:47:37.999 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_memory_target_ratio", 2024-03-26T06:47:37.999 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:37.999 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:37.999 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:37.999 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.7", 2024-03-26T06:47:37.999 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:37.999 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.000 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.000 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-26T06:47:38.000 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.000 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.001 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.001 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_log_destination": { 2024-03-26T06:47:38.002 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm_log_destination", 2024-03-26T06:47:38.002 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.002 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.002 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.002 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.002 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.002 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.002 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.003 INFO:teuthology.orchestra.run.smithi038.stdout: "file", 2024-03-26T06:47:38.003 INFO:teuthology.orchestra.run.smithi038.stdout: "file,syslog", 2024-03-26T06:47:38.003 INFO:teuthology.orchestra.run.smithi038.stdout: "syslog" 2024-03-26T06:47:38.003 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.003 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-26T06:47:38.003 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.003 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.003 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.004 INFO:teuthology.orchestra.run.smithi038.stdout: "cgroups_split": { 2024-03-26T06:47:38.004 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cgroups_split", 2024-03-26T06:47:38.004 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.004 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.004 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.004 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.004 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.004 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.005 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.005 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-26T06:47:38.005 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.005 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.005 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.005 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.005 INFO:teuthology.orchestra.run.smithi038.stdout: "config_checks_enabled": { 2024-03-26T06:47:38.005 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_checks_enabled", 2024-03-26T06:47:38.006 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.006 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.006 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.006 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.006 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.006 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.006 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.006 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-26T06:47:38.007 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.007 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.007 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.007 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.007 INFO:teuthology.orchestra.run.smithi038.stdout: "config_dashboard": { 2024-03-26T06:47:38.007 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_dashboard", 2024-03-26T06:47:38.007 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.007 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.008 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.008 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.008 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.008 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.008 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.008 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-26T06:47:38.008 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.008 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.009 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.009 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.009 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_alertmanager": { 2024-03-26T06:47:38.009 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_alertmanager", 2024-03-26T06:47:38.009 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.009 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.009 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.009 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-26T06:47:38.010 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.010 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.010 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.010 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:38.010 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.010 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.010 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.011 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_base": { 2024-03-26T06:47:38.011 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_base", 2024-03-26T06:47:38.011 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.011 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.011 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.011 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-26T06:47:38.011 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.011 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.012 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.012 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Container image name, without the tag", 2024-03-26T06:47:38.012 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.012 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.012 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.012 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.012 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_elasticsearch": { 2024-03-26T06:47:38.012 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_elasticsearch", 2024-03-26T06:47:38.012 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.013 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.013 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.013 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-26T06:47:38.013 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.013 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.013 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.013 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "elasticsearch container image", 2024-03-26T06:47:38.013 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.014 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.014 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.014 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.014 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_grafana": { 2024-03-26T06:47:38.014 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_grafana", 2024-03-26T06:47:38.014 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.014 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.014 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.015 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-26T06:47:38.015 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.015 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.015 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.015 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:38.015 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.015 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.015 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_haproxy": { 2024-03-26T06:47:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_haproxy", 2024-03-26T06:47:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-26T06:47:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.016 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.017 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.017 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "HAproxy container image", 2024-03-26T06:47:38.017 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.017 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.017 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.017 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.017 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_agent": { 2024-03-26T06:47:38.017 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_agent", 2024-03-26T06:47:38.018 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.018 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.018 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.018 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-26T06:47:38.018 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.018 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.018 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.018 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger agent container image", 2024-03-26T06:47:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_collector": { 2024-03-26T06:47:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_collector", 2024-03-26T06:47:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.019 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-26T06:47:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger collector container image", 2024-03-26T06:47:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.020 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_query": { 2024-03-26T06:47:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_query", 2024-03-26T06:47:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-26T06:47:38.021 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger query container image", 2024-03-26T06:47:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.022 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_keepalived": { 2024-03-26T06:47:38.023 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_keepalived", 2024-03-26T06:47:38.023 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.023 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.023 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.023 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-26T06:47:38.023 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.023 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.023 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.024 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Keepalived container image", 2024-03-26T06:47:38.024 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.024 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.024 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.024 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_loki": { 2024-03-26T06:47:38.024 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_loki", 2024-03-26T06:47:38.024 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.025 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.025 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.025 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-26T06:47:38.025 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.025 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.025 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.025 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Loki container image", 2024-03-26T06:47:38.025 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.026 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.026 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.026 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.026 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_node_exporter": { 2024-03-26T06:47:38.026 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_node_exporter", 2024-03-26T06:47:38.026 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.026 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.026 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.027 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-26T06:47:38.027 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.027 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.027 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.027 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:38.027 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.027 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.027 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.028 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.028 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_nvmeof": { 2024-03-26T06:47:38.028 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_nvmeof", 2024-03-26T06:47:38.028 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.028 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.028 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.028 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-26T06:47:38.028 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.029 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.029 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.029 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Nvme-of container image", 2024-03-26T06:47:38.029 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.029 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.029 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.029 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_prometheus": { 2024-03-26T06:47:38.029 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_prometheus", 2024-03-26T06:47:38.030 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.030 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.030 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.030 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-26T06:47:38.030 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.030 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.030 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.030 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:38.031 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.031 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.031 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.031 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_promtail": { 2024-03-26T06:47:38.031 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_promtail", 2024-03-26T06:47:38.031 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.031 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.032 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.032 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-26T06:47:38.032 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.032 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.032 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.032 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Promtail container image", 2024-03-26T06:47:38.032 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.032 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.032 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_snmp_gateway": { 2024-03-26T06:47:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_snmp_gateway", 2024-03-26T06:47:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-26T06:47:38.033 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SNMP Gateway container image", 2024-03-26T06:47:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.034 INFO:teuthology.orchestra.run.smithi038.stdout: "container_init": { 2024-03-26T06:47:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_init", 2024-03-26T06:47:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.035 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Run podman/docker with `--init`", 2024-03-26T06:47:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: "daemon_cache_timeout": { 2024-03-26T06:47:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "daemon_cache_timeout", 2024-03-26T06:47:38.036 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-26T06:47:38.037 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "default_cephadm_command_timeout": { 2024-03-26T06:47:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_cephadm_command_timeout", 2024-03-26T06:47:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.038 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "900", 2024-03-26T06:47:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-26T06:47:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "default_registry": { 2024-03-26T06:47:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_registry", 2024-03-26T06:47:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io", 2024-03-26T06:47:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.040 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-26T06:47:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: "device_cache_timeout": { 2024-03-26T06:47:38.041 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_cache_timeout", 2024-03-26T06:47:38.042 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.042 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.042 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.042 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1800", 2024-03-26T06:47:38.042 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.042 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.042 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.042 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache device inventory", 2024-03-26T06:47:38.042 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.043 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.043 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.043 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.043 INFO:teuthology.orchestra.run.smithi038.stdout: "device_enhanced_scan": { 2024-03-26T06:47:38.043 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_enhanced_scan", 2024-03-26T06:47:38.043 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.043 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.044 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.044 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.044 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.044 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.044 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.044 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-26T06:47:38.044 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.044 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.045 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.045 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.045 INFO:teuthology.orchestra.run.smithi038.stdout: "facts_cache_timeout": { 2024-03-26T06:47:38.045 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "facts_cache_timeout", 2024-03-26T06:47:38.045 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.045 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.045 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.045 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:38.045 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.046 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.046 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.046 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache host facts data", 2024-03-26T06:47:38.046 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.046 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.046 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.046 INFO:teuthology.orchestra.run.smithi038.stdout: "host_check_interval": { 2024-03-26T06:47:38.047 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "host_check_interval", 2024-03-26T06:47:38.047 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.047 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.047 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.047 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:38.047 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.047 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.047 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.048 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to perform a host check", 2024-03-26T06:47:38.048 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.048 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.048 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.048 INFO:teuthology.orchestra.run.smithi038.stdout: "hw_monitoring": { 2024-03-26T06:47:38.048 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hw_monitoring", 2024-03-26T06:47:38.048 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.049 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.049 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.049 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.049 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.049 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.049 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.049 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-26T06:47:38.049 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.050 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.050 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.050 INFO:teuthology.orchestra.run.smithi038.stdout: "inventory_list_all": { 2024-03-26T06:47:38.050 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "inventory_list_all", 2024-03-26T06:47:38.050 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.050 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.050 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.051 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.051 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.051 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.051 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.051 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-26T06:47:38.051 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.051 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.051 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.052 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.052 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.052 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.052 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.052 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.052 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.052 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.052 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.053 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.053 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.053 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.053 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.053 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.053 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.053 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.053 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.053 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.054 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.054 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.054 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.054 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.054 INFO:teuthology.orchestra.run.smithi038.stdout: "log_refresh_metadata": { 2024-03-26T06:47:38.054 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_refresh_metadata", 2024-03-26T06:47:38.054 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.055 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.055 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.055 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.055 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.055 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.055 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.055 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-26T06:47:38.055 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.056 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.056 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.056 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.056 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.056 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.056 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.056 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.056 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.057 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.057 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.057 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.057 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-26T06:47:38.057 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.057 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.057 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.058 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.058 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.058 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.058 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.058 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.058 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.058 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.059 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.059 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.059 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.059 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.059 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.059 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.059 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.059 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.060 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.060 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.060 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.060 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.060 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.060 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.060 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.060 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.061 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.061 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.061 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.061 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.061 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.061 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.061 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.061 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.062 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.062 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.062 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.062 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.062 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-26T06:47:38.062 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-26T06:47:38.062 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.062 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.063 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.063 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.063 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.063 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.063 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.063 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-26T06:47:38.063 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.063 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.063 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.064 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.064 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-26T06:47:38.064 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-26T06:47:38.064 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.064 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.064 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.064 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "*", 2024-03-26T06:47:38.064 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.065 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.065 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.065 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-26T06:47:38.065 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.065 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.065 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.065 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.065 INFO:teuthology.orchestra.run.smithi038.stdout: "max_count_per_host": { 2024-03-26T06:47:38.065 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_count_per_host", 2024-03-26T06:47:38.066 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.066 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.066 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.066 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.066 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.066 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.066 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.066 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of daemons per service per host", 2024-03-26T06:47:38.067 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.067 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.067 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.067 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.067 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd_draining_count": { 2024-03-26T06:47:38.067 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_osd_draining_count", 2024-03-26T06:47:38.067 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.067 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.068 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.068 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.068 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.068 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.068 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.068 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-26T06:47:38.068 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.068 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.069 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.069 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.069 INFO:teuthology.orchestra.run.smithi038.stdout: "migration_current": { 2024-03-26T06:47:38.069 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "migration_current", 2024-03-26T06:47:38.069 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.069 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.069 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.069 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.069 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.070 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.070 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.070 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "internal - do not modify", 2024-03-26T06:47:38.070 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.070 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.070 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.070 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-03-26T06:47:38.071 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-03-26T06:47:38.071 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.071 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.071 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.071 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "root", 2024-03-26T06:47:38.071 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.071 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.071 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.072 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm-package", 2024-03-26T06:47:38.072 INFO:teuthology.orchestra.run.smithi038.stdout: "root" 2024-03-26T06:47:38.072 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.072 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "mode for remote execution of cephadm", 2024-03-26T06:47:38.072 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.072 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.072 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.073 INFO:teuthology.orchestra.run.smithi038.stdout: "oob_default_addr": { 2024-03-26T06:47:38.073 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "oob_default_addr", 2024-03-26T06:47:38.073 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.073 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.073 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.073 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "169.254.1.1", 2024-03-26T06:47:38.073 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.073 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.074 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.074 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-26T06:47:38.074 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.074 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.074 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.074 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.074 INFO:teuthology.orchestra.run.smithi038.stdout: "prometheus_alerts_path": { 2024-03-26T06:47:38.074 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus_alerts_path", 2024-03-26T06:47:38.075 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.075 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.075 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.075 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-26T06:47:38.075 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.075 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.075 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.075 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-26T06:47:38.075 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.076 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.076 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.076 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.076 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_insecure": { 2024-03-26T06:47:38.076 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_insecure", 2024-03-26T06:47:38.076 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.076 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.076 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.077 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.077 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.077 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.077 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.077 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-26T06:47:38.077 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.077 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.077 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.078 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.078 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_password": { 2024-03-26T06:47:38.078 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_password", 2024-03-26T06:47:38.078 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.078 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.078 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.078 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.078 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.079 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.079 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.079 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-26T06:47:38.079 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.079 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.079 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.079 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.079 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_url": { 2024-03-26T06:47:38.080 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_url", 2024-03-26T06:47:38.080 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.080 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.080 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.080 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.080 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.080 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.080 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.081 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-26T06:47:38.081 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.081 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.081 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.081 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_username": { 2024-03-26T06:47:38.081 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_username", 2024-03-26T06:47:38.081 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.081 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.082 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.082 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.082 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.082 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.082 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.082 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-26T06:47:38.082 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.082 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.083 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.083 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.083 INFO:teuthology.orchestra.run.smithi038.stdout: "secure_monitoring_stack": { 2024-03-26T06:47:38.083 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "secure_monitoring_stack", 2024-03-26T06:47:38.083 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.083 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.083 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.083 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.084 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.084 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.084 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.084 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-26T06:47:38.084 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.084 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.084 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.084 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "service_discovery_port": { 2024-03-26T06:47:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "service_discovery_port", 2024-03-26T06:47:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8765", 2024-03-26T06:47:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.085 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.086 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "cephadm service discovery port", 2024-03-26T06:47:38.086 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.086 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.086 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.086 INFO:teuthology.orchestra.run.smithi038.stdout: "ssh_config_file": { 2024-03-26T06:47:38.086 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssh_config_file", 2024-03-26T06:47:38.086 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.087 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.087 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.087 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.087 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.087 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.087 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.087 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-26T06:47:38.087 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: "use_agent": { 2024-03-26T06:47:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_agent", 2024-03-26T06:47:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.088 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-26T06:47:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.090 INFO:teuthology.orchestra.run.smithi038.stdout: "use_repo_digest": { 2024-03-26T06:47:38.090 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_repo_digest", 2024-03-26T06:47:38.090 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.090 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.090 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.090 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.090 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.090 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-26T06:47:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_failed_host_check": { 2024-03-26T06:47:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_failed_host_check", 2024-03-26T06:47:38.091 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.092 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.092 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.092 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.092 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.092 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.092 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.092 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if the host check fails", 2024-03-26T06:47:38.092 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_daemons": { 2024-03-26T06:47:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_daemons", 2024-03-26T06:47:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.093 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.094 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.094 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.094 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.094 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-26T06:47:38.094 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.094 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.094 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.094 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_hosts": { 2024-03-26T06:47:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_hosts", 2024-03-26T06:47:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.095 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-26T06:47:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.096 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.097 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crash", 2024-03-26T06:47:38.097 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.097 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.097 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.097 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.097 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.097 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.098 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.098 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.098 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.098 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.098 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.098 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.098 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.098 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.099 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.099 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.099 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.099 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.099 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.099 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.099 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.099 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.100 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.100 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.100 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.100 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.100 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.100 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.100 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.100 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.101 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.102 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.102 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.102 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.102 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.102 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.102 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.102 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.102 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.102 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.103 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.103 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.103 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.103 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.103 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.103 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.103 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.104 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.104 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.104 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.104 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.104 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.104 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.104 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.104 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.105 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.105 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.105 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.105 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.105 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.105 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.105 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.105 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.106 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.106 INFO:teuthology.orchestra.run.smithi038.stdout: "retain_interval": { 2024-03-26T06:47:38.106 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retain_interval", 2024-03-26T06:47:38.106 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.106 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.106 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.106 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "31536000", 2024-03-26T06:47:38.106 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.107 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.107 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.107 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-26T06:47:38.107 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.107 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.107 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.107 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_recent_interval": { 2024-03-26T06:47:38.108 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_recent_interval", 2024-03-26T06:47:38.108 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.108 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.108 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.108 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1209600", 2024-03-26T06:47:38.108 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.108 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.108 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.109 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-26T06:47:38.109 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.109 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.109 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.109 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.109 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.109 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dashboard", 2024-03-26T06:47:38.110 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.110 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.110 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.110 INFO:teuthology.orchestra.run.smithi038.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-26T06:47:38.110 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-26T06:47:38.110 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.110 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.110 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.111 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.111 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.111 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.111 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.111 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.111 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.111 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.111 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.112 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_HOST": { 2024-03-26T06:47:38.112 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-26T06:47:38.112 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.112 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.112 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.112 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.112 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.112 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.113 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.113 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.113 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.113 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.113 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.113 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-26T06:47:38.113 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-26T06:47:38.113 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.114 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_ENABLED": { 2024-03-26T06:47:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_ENABLED", 2024-03-26T06:47:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.115 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.117 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-26T06:47:38.117 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-26T06:47:38.117 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.117 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.117 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.117 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.117 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.117 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.118 INFO:teuthology.orchestra.run.smithi038.stdout: "ENABLE_BROWSABLE_API": { 2024-03-26T06:47:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-26T06:47:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.119 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-26T06:47:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-26T06:47:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.120 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.121 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-26T06:47:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-26T06:47:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.122 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-26T06:47:38.123 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-26T06:47:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.124 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-26T06:47:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-26T06:47:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.125 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.126 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.127 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.127 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-26T06:47:38.127 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-26T06:47:38.127 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.127 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.127 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.127 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.128 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.128 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.128 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.128 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.128 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.128 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.128 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.128 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.128 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-26T06:47:38.129 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-26T06:47:38.129 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.129 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.129 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.129 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.129 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.129 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.129 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.130 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.130 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.130 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.130 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.130 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.130 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-26T06:47:38.130 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-26T06:47:38.130 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.131 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.131 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.131 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.131 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.131 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.131 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.131 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.131 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.132 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.132 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.132 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.132 INFO:teuthology.orchestra.run.smithi038.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-26T06:47:38.132 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-26T06:47:38.132 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.132 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.132 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.132 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.133 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.133 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.133 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.133 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.133 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.133 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.133 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.134 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_PASSWORD": { 2024-03-26T06:47:38.134 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-26T06:47:38.134 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.134 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.134 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.134 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-03-26T06:47:38.134 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.134 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.135 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.135 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.135 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.135 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.135 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.135 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.135 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-26T06:47:38.135 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-26T06:47:38.135 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.136 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.136 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.136 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.136 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.136 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.136 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.136 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.136 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.137 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.137 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.137 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_URL": { 2024-03-26T06:47:38.137 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_URL", 2024-03-26T06:47:38.137 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.137 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.137 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.138 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.138 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.138 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.138 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.138 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.138 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.138 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.138 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.139 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_USERNAME": { 2024-03-26T06:47:38.139 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-26T06:47:38.139 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.139 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.139 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.139 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-03-26T06:47:38.139 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.139 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.140 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.140 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.140 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.140 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.140 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.140 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.140 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-26T06:47:38.140 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-26T06:47:38.141 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.141 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.141 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.141 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.141 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.141 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.141 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.141 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-26T06:47:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-26T06:47:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.142 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.143 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.143 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.143 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.143 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.143 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.143 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.143 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.143 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.143 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.144 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.144 INFO:teuthology.orchestra.run.smithi038.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-26T06:47:38.144 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-26T06:47:38.144 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.144 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.144 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.144 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.144 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.145 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.145 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.145 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.145 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.145 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.145 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.145 INFO:teuthology.orchestra.run.smithi038.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-26T06:47:38.145 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-26T06:47:38.146 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.146 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.146 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.146 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.146 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.146 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.146 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.147 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.147 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.147 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.147 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.147 INFO:teuthology.orchestra.run.smithi038.stdout: "MULTICLUSTER_CONFIG": { 2024-03-26T06:47:38.147 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-26T06:47:38.147 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.148 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.148 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.148 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "{}", 2024-03-26T06:47:38.148 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.148 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.148 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.148 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.148 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.149 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.149 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.149 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_HOST": { 2024-03-26T06:47:38.149 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-26T06:47:38.149 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.149 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.149 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.149 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.150 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.150 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.150 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.150 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.150 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.150 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.150 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.151 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-26T06:47:38.151 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-26T06:47:38.151 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.151 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.151 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.151 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.151 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.151 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.152 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.152 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.152 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.152 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.152 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.152 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.152 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-26T06:47:38.152 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-26T06:47:38.153 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.153 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.153 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.153 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.153 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.153 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.153 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.153 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.153 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.154 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.154 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.154 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-26T06:47:38.154 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-26T06:47:38.154 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.154 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.154 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.155 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.155 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.155 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.155 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.155 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.155 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.155 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.155 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.156 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.156 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-26T06:47:38.156 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-26T06:47:38.156 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.156 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.156 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.156 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.156 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.156 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.157 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.157 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.157 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.157 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.157 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.157 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-26T06:47:38.157 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-26T06:47:38.158 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.158 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.158 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.158 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.158 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.158 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.158 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.158 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.159 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.159 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.159 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.159 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-26T06:47:38.159 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-26T06:47:38.159 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.159 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.159 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.160 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.160 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.160 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.160 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.160 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.160 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.160 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.160 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.161 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-26T06:47:38.161 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-26T06:47:38.161 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.161 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.161 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.161 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.161 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.161 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.162 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.162 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.162 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.162 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.162 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.162 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.162 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-26T06:47:38.162 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-26T06:47:38.163 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.163 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.163 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.163 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.163 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.163 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.163 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.163 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.164 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.164 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.164 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.164 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.164 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_ENABLED": { 2024-03-26T06:47:38.164 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-26T06:47:38.164 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.164 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.164 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.165 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.165 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.165 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.165 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.165 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.165 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.165 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.165 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.166 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-26T06:47:38.166 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-26T06:47:38.166 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.166 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.166 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.166 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-26T06:47:38.166 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.167 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.167 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.167 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.167 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.167 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.167 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.167 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.167 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-26T06:47:38.168 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-26T06:47:38.168 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.168 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.168 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.168 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.168 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.168 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.168 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.168 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.169 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.169 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.169 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.169 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.169 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-26T06:47:38.169 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-26T06:47:38.169 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.169 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.170 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.170 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8", 2024-03-26T06:47:38.170 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.170 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.170 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.170 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.170 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.170 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.171 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.171 INFO:teuthology.orchestra.run.smithi038.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-26T06:47:38.171 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-26T06:47:38.171 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.171 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.171 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.171 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "45", 2024-03-26T06:47:38.172 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.172 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.172 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.172 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.172 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.172 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.172 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.172 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.172 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ACCESS_KEY": { 2024-03-26T06:47:38.173 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-26T06:47:38.173 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.173 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.173 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.173 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.173 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.173 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.173 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.174 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.174 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.174 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.174 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.174 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.174 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-26T06:47:38.174 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-26T06:47:38.174 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.175 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.175 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.175 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-03-26T06:47:38.175 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.175 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.175 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.175 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.175 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.175 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.176 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.176 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.176 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SECRET_KEY": { 2024-03-26T06:47:38.176 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-26T06:47:38.176 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.176 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.176 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.176 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.177 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.177 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.177 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.177 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.177 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.177 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.177 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.178 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SSL_VERIFY": { 2024-03-26T06:47:38.178 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-26T06:47:38.178 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.178 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.178 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.178 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.178 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.178 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.179 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.179 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.179 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.179 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.179 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.179 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.179 INFO:teuthology.orchestra.run.smithi038.stdout: "UNSAFE_TLS_v1_2": { 2024-03-26T06:47:38.179 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-26T06:47:38.179 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.180 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.180 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.180 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.180 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.180 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.180 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.180 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.180 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.181 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.181 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.181 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.181 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-26T06:47:38.181 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-26T06:47:38.181 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.181 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.181 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.181 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:38.182 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.182 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.182 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.182 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.182 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.182 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.182 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.183 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-26T06:47:38.183 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-26T06:47:38.183 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.183 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.183 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.183 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.183 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.183 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.184 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.184 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.184 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.184 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.184 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.184 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-26T06:47:38.184 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-26T06:47:38.185 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.185 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.185 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.185 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:38.185 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.185 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.185 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.185 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.185 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.186 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.186 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.186 INFO:teuthology.orchestra.run.smithi038.stdout: "cross_origin_url": { 2024-03-26T06:47:38.186 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cross_origin_url", 2024-03-26T06:47:38.186 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.186 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.186 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.187 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.187 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.187 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.187 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.187 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.187 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.187 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.187 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.188 INFO:teuthology.orchestra.run.smithi038.stdout: "crt_file": { 2024-03-26T06:47:38.188 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crt_file", 2024-03-26T06:47:38.188 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.188 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.188 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.188 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.188 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.189 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.189 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.189 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.189 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.189 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.189 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.189 INFO:teuthology.orchestra.run.smithi038.stdout: "debug": { 2024-03-26T06:47:38.189 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "debug", 2024-03-26T06:47:38.190 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.190 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.190 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.190 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.190 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.190 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.190 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.190 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable/disable debug options", 2024-03-26T06:47:38.191 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.191 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.191 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.191 INFO:teuthology.orchestra.run.smithi038.stdout: "jwt_token_ttl": { 2024-03-26T06:47:38.191 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "jwt_token_ttl", 2024-03-26T06:47:38.191 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.191 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.192 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.192 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "28800", 2024-03-26T06:47:38.192 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.192 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.192 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.192 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.192 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.192 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.192 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.193 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-03-26T06:47:38.193 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-03-26T06:47:38.193 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.193 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.193 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.193 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.193 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.194 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.194 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.194 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.194 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.194 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.194 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.194 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.194 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.195 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.195 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.195 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.195 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.195 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.195 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.195 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.195 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.195 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.196 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.196 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.196 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.196 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.196 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.196 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.196 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.196 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.197 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.197 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.197 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.197 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.197 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.197 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.197 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.197 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.198 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.198 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.198 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.198 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.198 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.198 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.198 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.198 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.199 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.199 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.199 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.199 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.199 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.199 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.199 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.199 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.200 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.200 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.200 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.200 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.200 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.200 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.200 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.200 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.201 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.201 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.201 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.201 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.201 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.201 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.201 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.201 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.202 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.202 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.202 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.202 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.202 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.202 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.202 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.202 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.203 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.203 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.203 INFO:teuthology.orchestra.run.smithi038.stdout: "motd": { 2024-03-26T06:47:38.203 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "motd", 2024-03-26T06:47:38.203 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.203 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.203 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.204 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.204 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.204 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.204 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.204 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "The message of the day", 2024-03-26T06:47:38.204 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.204 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.204 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.204 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.205 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_resolve_ip_addr": { 2024-03-26T06:47:38.205 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "redirect_resolve_ip_addr", 2024-03-26T06:47:38.205 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.205 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.205 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.205 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.205 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.205 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.206 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.206 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.206 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.206 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.206 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.206 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.206 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-03-26T06:47:38.206 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-03-26T06:47:38.207 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.207 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.207 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.207 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-03-26T06:47:38.207 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.207 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.207 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.209 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.209 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.210 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.210 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.210 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-03-26T06:47:38.210 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-03-26T06:47:38.210 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.210 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.210 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.211 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8080", 2024-03-26T06:47:38.211 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.211 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.211 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.211 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.211 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.211 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.211 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.211 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.212 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-03-26T06:47:38.212 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-03-26T06:47:38.212 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.212 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.212 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.212 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.212 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.212 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.213 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.213 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.213 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.213 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.213 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.213 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.213 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl_server_port": { 2024-03-26T06:47:38.213 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl_server_port", 2024-03-26T06:47:38.213 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.214 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.214 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.214 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8443", 2024-03-26T06:47:38.214 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.214 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.214 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.214 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.214 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.215 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.215 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.215 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.215 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-03-26T06:47:38.215 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-03-26T06:47:38.215 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.215 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.215 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.216 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "redirect", 2024-03-26T06:47:38.216 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.216 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.216 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.216 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.216 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect" 2024-03-26T06:47:38.216 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.216 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.217 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.217 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.217 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.217 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.217 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-03-26T06:47:38.217 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-03-26T06:47:38.217 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.217 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.218 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.218 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-03-26T06:47:38.218 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-03-26T06:47:38.218 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-03-26T06:47:38.218 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.218 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.218 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.218 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.219 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.219 INFO:teuthology.orchestra.run.smithi038.stdout: "url_prefix": { 2024-03-26T06:47:38.219 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url_prefix", 2024-03-26T06:47:38.219 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.219 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.219 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.219 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.219 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.220 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.220 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.220 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.220 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.220 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.220 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.220 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.220 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.221 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.221 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-03-26T06:47:38.221 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.221 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.221 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.221 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_monitoring": { 2024-03-26T06:47:38.221 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_monitoring", 2024-03-26T06:47:38.222 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.222 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.222 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.222 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.222 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.222 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.222 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.222 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "monitor device health metrics", 2024-03-26T06:47:38.223 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.223 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.223 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.223 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.223 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.223 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.223 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.223 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.224 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.224 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.224 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.224 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.224 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.224 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.224 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.224 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.225 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.225 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.225 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.225 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.225 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.225 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.225 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.225 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.226 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.226 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.226 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.226 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.226 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.226 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.226 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.227 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.227 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.227 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.227 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.227 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.227 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.227 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.228 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.228 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.228 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.228 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.228 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.228 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.228 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.228 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.229 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.229 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.229 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.229 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.229 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.229 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.229 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.229 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.229 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.230 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.230 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.230 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.230 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.230 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.230 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.230 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.231 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.231 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.231 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.231 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.231 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.231 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.231 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.231 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.232 INFO:teuthology.orchestra.run.smithi038.stdout: "mark_out_threshold": { 2024-03-26T06:47:38.232 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mark_out_threshold", 2024-03-26T06:47:38.232 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.232 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.232 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.232 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2419200", 2024-03-26T06:47:38.232 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.232 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.232 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.233 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-26T06:47:38.233 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.233 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.233 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.233 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.233 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": { 2024-03-26T06:47:38.233 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_name", 2024-03-26T06:47:38.233 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.234 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.234 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.234 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "device_health_metrics", 2024-03-26T06:47:38.234 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.234 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.234 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.234 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-26T06:47:38.234 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.235 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.235 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.235 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.235 INFO:teuthology.orchestra.run.smithi038.stdout: "retention_period": { 2024-03-26T06:47:38.235 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retention_period", 2024-03-26T06:47:38.235 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.235 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.235 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.235 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15552000", 2024-03-26T06:47:38.236 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.236 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.236 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.236 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain device health metrics", 2024-03-26T06:47:38.236 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.236 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.237 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_frequency": { 2024-03-26T06:47:38.237 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_frequency", 2024-03-26T06:47:38.237 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.237 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.237 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.237 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-03-26T06:47:38.237 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.237 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.238 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.238 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-26T06:47:38.238 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.238 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.238 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.238 INFO:teuthology.orchestra.run.smithi038.stdout: "self_heal": { 2024-03-26T06:47:38.238 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "self_heal", 2024-03-26T06:47:38.239 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.239 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.239 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.239 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.239 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.239 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.239 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.239 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-26T06:47:38.239 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.240 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 -- 172.21.15.38:0/1047302382 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fee340a2970 msgr2=0x7fee340a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:38.240 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fee340a2970 0x7fee340a4e30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fee4c0ffd40 tx=0x7fee40005fd0 comp rx=0 tx=0).stop 2024-03-26T06:47:38.241 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.241 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:38.241 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:38.241 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.241 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.241 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.242 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:38.242 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.242 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.242 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.242 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and check device health", 2024-03-26T06:47:38.242 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.242 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.242 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.243 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_threshold": { 2024-03-26T06:47:38.243 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_threshold", 2024-03-26T06:47:38.243 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.243 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.243 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.243 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7257600", 2024-03-26T06:47:38.243 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.243 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.244 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.244 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-26T06:47:38.244 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.244 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.244 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.244 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.245 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.245 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "diskprediction_local", 2024-03-26T06:47:38.245 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.245 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.245 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.245 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.245 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.245 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.246 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.246 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.246 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.246 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.246 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.246 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.246 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.246 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.247 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.247 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.247 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.247 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.247 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.247 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.247 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.247 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.248 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.248 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.248 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.248 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.248 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.248 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.248 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.249 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.249 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.249 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.249 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.249 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.249 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.249 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.249 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.250 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.250 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.250 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.250 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.250 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.250 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.250 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.250 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.251 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.251 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.251 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.251 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.251 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.251 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.251 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.251 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.252 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.252 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.252 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.252 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.252 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.252 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.252 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.252 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.253 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.253 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.253 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.253 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.253 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.253 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.253 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.253 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.253 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.254 INFO:teuthology.orchestra.run.smithi038.stdout: "predict_interval": { 2024-03-26T06:47:38.254 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predict_interval", 2024-03-26T06:47:38.254 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.254 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.254 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.254 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-03-26T06:47:38.254 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.254 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.255 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.255 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.255 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.255 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.255 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.255 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.255 INFO:teuthology.orchestra.run.smithi038.stdout: "predictor_model": { 2024-03-26T06:47:38.255 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predictor_model", 2024-03-26T06:47:38.255 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.256 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.256 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.256 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "prophetstor", 2024-03-26T06:47:38.256 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.256 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.256 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.256 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.256 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.257 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.257 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.257 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.257 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:38.257 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:38.257 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.257 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.258 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.258 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:38.258 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.258 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.258 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.258 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.258 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.258 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.259 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.259 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.259 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.259 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.259 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "influx", 2024-03-26T06:47:38.259 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": false, 2024-03-26T06:47:38.259 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "influxdb python module not found", 2024-03-26T06:47:38.259 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.260 INFO:teuthology.orchestra.run.smithi038.stdout: "batch_size": { 2024-03-26T06:47:38.260 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "batch_size", 2024-03-26T06:47:38.260 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.260 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.260 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.260 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5000", 2024-03-26T06:47:38.260 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.260 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.261 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.261 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-26T06:47:38.261 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.261 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.261 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.261 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.261 INFO:teuthology.orchestra.run.smithi038.stdout: "database": { 2024-03-26T06:47:38.261 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "database", 2024-03-26T06:47:38.261 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.262 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.262 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.262 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "ceph", 2024-03-26T06:47:38.262 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.262 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.262 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.262 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-26T06:47:38.262 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.263 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.263 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.263 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.263 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": { 2024-03-26T06:47:38.263 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hostname", 2024-03-26T06:47:38.263 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.263 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.263 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.263 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.264 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.264 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.264 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.264 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server hostname", 2024-03-26T06:47:38.264 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.264 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.264 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.265 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:38.265 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:38.265 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.265 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.265 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.265 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "30", 2024-03-26T06:47:38.265 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "5", 2024-03-26T06:47:38.265 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.266 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.266 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-26T06:47:38.266 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.266 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.266 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.266 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.266 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.266 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.267 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.267 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.267 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.267 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.267 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.267 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.267 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.267 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.268 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.268 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.268 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.268 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.268 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.268 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.268 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.268 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.268 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.269 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.269 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.269 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.269 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.269 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.269 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.269 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.270 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.270 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.270 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.270 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.270 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.270 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.270 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.270 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.271 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.271 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.271 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.271 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.271 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.271 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.271 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.271 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.271 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.272 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.272 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.272 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.272 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.272 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.272 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.272 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.272 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.272 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.273 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.273 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.273 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.273 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.273 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.273 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.273 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.274 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.274 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.274 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.274 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.274 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.274 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.274 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.274 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.274 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.275 INFO:teuthology.orchestra.run.smithi038.stdout: "password": { 2024-03-26T06:47:38.275 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "password", 2024-03-26T06:47:38.275 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.275 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.275 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.275 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.275 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.275 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.276 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.276 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "password of InfluxDB server user", 2024-03-26T06:47:38.276 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.276 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.276 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.276 INFO:teuthology.orchestra.run.smithi038.stdout: "port": { 2024-03-26T06:47:38.276 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "port", 2024-03-26T06:47:38.277 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.277 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.277 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.277 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8086", 2024-03-26T06:47:38.277 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.277 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.277 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.277 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server port", 2024-03-26T06:47:38.278 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.278 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.278 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.278 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-03-26T06:47:38.278 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-03-26T06:47:38.278 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.278 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.279 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.279 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "false", 2024-03-26T06:47:38.279 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.279 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.279 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.279 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-26T06:47:38.279 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.279 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.279 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.280 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.280 INFO:teuthology.orchestra.run.smithi038.stdout: "threads": { 2024-03-26T06:47:38.280 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threads", 2024-03-26T06:47:38.280 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.280 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.280 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.280 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:38.280 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "32", 2024-03-26T06:47:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-26T06:47:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: "username": { 2024-03-26T06:47:38.281 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "username", 2024-03-26T06:47:38.282 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.282 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.282 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.282 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.282 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.282 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.282 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.282 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "username of InfluxDB server user", 2024-03-26T06:47:38.283 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.283 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.283 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.283 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.283 INFO:teuthology.orchestra.run.smithi038.stdout: "verify_ssl": { 2024-03-26T06:47:38.283 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "verify_ssl", 2024-03-26T06:47:38.283 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.283 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.283 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.284 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "true", 2024-03-26T06:47:38.284 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.284 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.284 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.284 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-26T06:47:38.284 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.284 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.284 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.285 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.285 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.285 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "insights", 2024-03-26T06:47:38.285 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.285 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.285 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.286 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.286 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.286 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.286 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.286 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.286 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.286 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.286 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.287 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.287 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.287 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.287 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.287 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.287 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.287 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.287 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.288 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.288 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.288 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.288 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.288 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.288 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.288 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.288 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.289 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.289 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.289 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.289 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.289 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.289 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.289 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.289 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.289 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.290 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.290 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.290 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.290 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.290 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.290 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.290 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.290 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.291 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.291 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.291 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.291 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.291 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.291 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.291 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.291 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.291 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.292 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.292 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.292 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.292 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.292 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.292 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.292 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.292 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.292 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.293 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.293 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.293 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.293 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.293 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.293 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.293 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.293 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.294 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.294 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.294 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.294 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.294 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.294 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iostat", 2024-03-26T06:47:38.294 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.294 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.295 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.295 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.295 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.295 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.295 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.295 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.295 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.295 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.296 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.296 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.296 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.296 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.296 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.296 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.296 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.296 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.297 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.297 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.297 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.297 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.297 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.297 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.297 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.298 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.298 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.298 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.298 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.298 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.298 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.298 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.298 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.299 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.299 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.299 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.299 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.299 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.299 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.299 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.300 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.300 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.300 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.300 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.300 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.300 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.300 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.300 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.300 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.301 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.301 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.301 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.301 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.301 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.301 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.301 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.302 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.302 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.302 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.302 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.302 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.302 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.302 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.302 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.302 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.303 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.303 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.303 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.303 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.303 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.304 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.304 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "k8sevents", 2024-03-26T06:47:38.304 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.304 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.304 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.304 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_event_retention_days": { 2024-03-26T06:47:38.304 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ceph_event_retention_days", 2024-03-26T06:47:38.304 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.304 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.305 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.305 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7", 2024-03-26T06:47:38.305 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.305 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.305 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.305 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-26T06:47:38.305 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.305 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.306 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.306 INFO:teuthology.orchestra.run.smithi038.stdout: "config_check_secs": { 2024-03-26T06:47:38.306 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_check_secs", 2024-03-26T06:47:38.306 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.306 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.306 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.306 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.307 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "10", 2024-03-26T06:47:38.307 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.307 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.307 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-26T06:47:38.307 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.307 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.307 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.308 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.308 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.308 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.308 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.308 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.308 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.308 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.308 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.309 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.309 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.309 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.309 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.309 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.309 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.309 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.309 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.310 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.310 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.310 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.310 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.310 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.310 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.310 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.311 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.311 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.311 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.311 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.311 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.311 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.311 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.311 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.312 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.312 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.312 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.312 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.312 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.312 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.312 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.312 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.312 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.313 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.313 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.313 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.313 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.313 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.313 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.313 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.313 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.313 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.314 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.314 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.314 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.314 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.314 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.314 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.314 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.315 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.315 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.315 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.315 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.315 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.315 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.315 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.315 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.315 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.316 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.316 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.316 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.316 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.316 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "localpool", 2024-03-26T06:47:38.316 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.317 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.317 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.317 INFO:teuthology.orchestra.run.smithi038.stdout: "failure_domain": { 2024-03-26T06:47:38.317 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "failure_domain", 2024-03-26T06:47:38.317 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.317 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.317 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.317 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "host", 2024-03-26T06:47:38.318 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.318 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.318 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.318 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "failure domain for any created local pool", 2024-03-26T06:47:38.318 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-26T06:47:38.318 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.318 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.319 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.319 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.319 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.319 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.319 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.319 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.319 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.319 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.320 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.320 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.320 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.320 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.320 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.320 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.320 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.320 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.321 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.321 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.321 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.321 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.321 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.321 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.321 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.321 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.322 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.322 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.322 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.322 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.322 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.322 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.322 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.323 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.323 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.323 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.323 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.323 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.323 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.323 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.323 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.323 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.324 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.324 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.324 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.324 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.324 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.324 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.324 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.324 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.325 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.325 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.325 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.325 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.325 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.325 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.325 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.326 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.326 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.326 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.326 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.326 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.326 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.326 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.326 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.327 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.327 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.327 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.327 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.327 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": { 2024-03-26T06:47:38.327 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_size", 2024-03-26T06:47:38.327 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.327 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.328 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.328 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.328 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.328 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.328 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.328 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default min_size for any created local pool", 2024-03-26T06:47:38.328 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-26T06:47:38.328 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.328 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.329 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.329 INFO:teuthology.orchestra.run.smithi038.stdout: "num_rep": { 2024-03-26T06:47:38.329 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "num_rep", 2024-03-26T06:47:38.329 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.329 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.329 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.329 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3", 2024-03-26T06:47:38.329 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.330 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.330 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.330 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default replica count for any created local pool", 2024-03-26T06:47:38.330 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.330 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.330 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.330 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": { 2024-03-26T06:47:38.331 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_num", 2024-03-26T06:47:38.331 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.331 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.331 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.331 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "128", 2024-03-26T06:47:38.331 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.331 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.331 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.332 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default pg_num for any created local pool", 2024-03-26T06:47:38.332 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.332 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.332 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 -- 172.21.15.38:0/1047302382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c069c10 msgr2=0x7fee4c19d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:47:38.333 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c069c10 0x7fee4c19d870 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fee3c00b4a0 tx=0x7fee3c00b960 comp rx=0 tx=0).stop 2024-03-26T06:47:38.333 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.333 INFO:teuthology.orchestra.run.smithi038.stdout: "prefix": { 2024-03-26T06:47:38.333 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prefix", 2024-03-26T06:47:38.334 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.334 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.334 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.334 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.334 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.334 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.334 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.334 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name prefix for any created local pool", 2024-03-26T06:47:38.335 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.335 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.335 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.335 INFO:teuthology.orchestra.run.smithi038.stdout: "subtree": { 2024-03-26T06:47:38.335 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "subtree", 2024-03-26T06:47:38.335 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.335 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.335 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.336 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "rack", 2024-03-26T06:47:38.336 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.336 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.336 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.336 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-26T06:47:38.336 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-26T06:47:38.336 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.336 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.337 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.337 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.337 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.337 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mds_autoscaler", 2024-03-26T06:47:38.337 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.337 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.337 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.338 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.338 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.338 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.338 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.338 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.338 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.338 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.338 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.339 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.340 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.341 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.342 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.343 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.344 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.345 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.346 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirroring", 2024-03-26T06:47:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.347 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.348 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.350 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.351 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.352 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.353 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.353 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.353 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.353 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.353 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.353 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.353 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.354 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.354 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.354 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.354 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.354 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.354 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.354 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.354 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.355 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.355 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.355 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.355 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.355 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.355 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.355 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.355 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.356 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.356 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.356 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.356 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "nfs", 2024-03-26T06:47:38.356 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.356 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.357 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.357 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.357 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.357 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.357 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.357 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.357 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.357 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.358 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.358 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.358 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.358 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.358 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.358 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.358 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.358 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.359 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.359 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.359 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.359 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.359 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.359 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.359 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.359 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.360 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.360 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.360 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.360 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.360 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.360 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.360 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.360 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.360 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.361 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.361 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.361 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.361 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.361 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.361 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.361 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.361 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.361 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.362 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.362 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.362 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.362 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.362 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.362 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.362 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.362 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.363 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.363 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.363 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.363 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.363 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.363 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.363 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.363 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.363 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.364 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.364 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.364 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.364 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.364 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.364 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.364 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.364 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.365 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.365 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.365 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.365 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.365 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.365 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.365 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.365 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-03-26T06:47:38.365 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.366 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.366 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.366 INFO:teuthology.orchestra.run.smithi038.stdout: "fail_fs": { 2024-03-26T06:47:38.366 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "fail_fs", 2024-03-26T06:47:38.366 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.366 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.366 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.366 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.367 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.367 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.367 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.367 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-26T06:47:38.367 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.367 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.367 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.367 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.368 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.368 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.368 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.368 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.368 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.368 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.368 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.368 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.369 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.369 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.369 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.369 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.369 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.369 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.369 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.369 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.369 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.370 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.370 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.370 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.370 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.370 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.370 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.370 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.371 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.371 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.371 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.371 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.371 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.371 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.371 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.371 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.372 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.372 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.372 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.372 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.372 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.372 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.372 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.372 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.372 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.373 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.373 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.373 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.373 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.373 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.373 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.373 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.373 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.373 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.374 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.374 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.374 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.374 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.374 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.374 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.374 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.375 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.375 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.375 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.375 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.375 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.375 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.375 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.375 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.376 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.376 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.376 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator": { 2024-03-26T06:47:38.376 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-03-26T06:47:38.376 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.376 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.376 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.376 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.377 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.377 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.377 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.377 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-03-26T06:47:38.377 INFO:teuthology.orchestra.run.smithi038.stdout: "rook", 2024-03-26T06:47:38.377 INFO:teuthology.orchestra.run.smithi038.stdout: "test_orchestrator" 2024-03-26T06:47:38.377 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.377 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Orchestrator backend", 2024-03-26T06:47:38.378 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.378 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.378 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.378 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.378 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.378 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_perf_query", 2024-03-26T06:47:38.379 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.379 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.379 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.379 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.379 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.379 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.379 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.379 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.379 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.380 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.380 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.380 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.380 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.380 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.380 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.380 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.380 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.380 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.381 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.381 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.381 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.381 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.381 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.381 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.381 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.381 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.382 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.382 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.382 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.382 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.382 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.382 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.382 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.382 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.383 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.383 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.383 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.383 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.383 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.383 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.383 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.383 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.383 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.384 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.384 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.384 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.384 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.384 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.384 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.384 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.384 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.384 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.385 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.385 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.385 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.385 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.385 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.385 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.385 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.385 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.385 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.386 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.387 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_support", 2024-03-26T06:47:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.388 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.389 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.390 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.391 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.392 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.393 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.394 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.394 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.394 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.394 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.394 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.394 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.394 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.395 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.396 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.397 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_autoscaler", 2024-03-26T06:47:38.397 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.397 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.397 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.397 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.397 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.397 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.398 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.398 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.398 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.398 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.398 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.398 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.398 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.398 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.399 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.399 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.399 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.399 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.399 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.399 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.399 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.399 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.399 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.400 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.400 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.400 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.400 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.400 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.400 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.400 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.401 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.401 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.401 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.401 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.401 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.401 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.401 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.401 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.401 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.402 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.402 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.402 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.402 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.402 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.402 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.402 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.402 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.403 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.403 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.403 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.403 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.403 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.403 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.403 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.403 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.403 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.404 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.404 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.404 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.404 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.404 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.404 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.404 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.404 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.405 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.405 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.405 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.405 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.405 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.405 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.405 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.405 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.406 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.406 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:38.406 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:38.406 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.406 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.406 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.406 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:38.406 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.407 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.407 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.407 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.407 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.407 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.407 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.407 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.407 INFO:teuthology.orchestra.run.smithi038.stdout: "threshold": { 2024-03-26T06:47:38.408 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threshold", 2024-03-26T06:47:38.408 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:38.408 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.408 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.408 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-03-26T06:47:38.408 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1.0", 2024-03-26T06:47:38.408 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.408 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.408 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "scaling threshold", 2024-03-26T06:47:38.409 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-26T06:47:38.409 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.409 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.409 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.409 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.409 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.409 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "progress", 2024-03-26T06:47:38.410 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.410 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.410 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.410 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_pg_recovery_event": { 2024-03-26T06:47:38.410 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_pg_recovery_event", 2024-03-26T06:47:38.410 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.410 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.410 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.411 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.411 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.411 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.411 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.411 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-26T06:47:38.411 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.411 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.411 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.411 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.412 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-03-26T06:47:38.412 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-03-26T06:47:38.412 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.412 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.413 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.414 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.414 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.414 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.414 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.414 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.414 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.414 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.414 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.415 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.415 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.415 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.415 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.415 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.415 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.415 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.415 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.415 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.416 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.416 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.416 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.416 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.416 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.416 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.416 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.416 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.417 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.417 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.417 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.417 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.417 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.417 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.417 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.417 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.418 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.418 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.418 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.418 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.418 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.418 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.418 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.418 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.419 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.419 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.419 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.419 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.419 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.419 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.419 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.419 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.419 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.420 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.420 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.420 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.420 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.420 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.420 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.420 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.420 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.421 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.421 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.421 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.421 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.421 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.421 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.421 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.421 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.421 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.422 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.422 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.422 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.422 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.422 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.422 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.422 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.422 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.423 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.423 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.423 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.423 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.423 INFO:teuthology.orchestra.run.smithi038.stdout: "max_completed_events": { 2024-03-26T06:47:38.423 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_completed_events", 2024-03-26T06:47:38.423 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.423 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.423 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.424 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "50", 2024-03-26T06:47:38.424 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.424 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.424 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.424 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "number of past completed events to remember", 2024-03-26T06:47:38.424 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.424 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.424 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.424 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.425 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:38.425 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:38.425 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.425 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.425 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.425 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:38.425 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.425 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.426 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.426 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long the module is going to sleep", 2024-03-26T06:47:38.426 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.426 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.426 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.426 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.426 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.427 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.427 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus", 2024-03-26T06:47:38.427 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.427 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.427 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.427 INFO:teuthology.orchestra.run.smithi038.stdout: "cache": { 2024-03-26T06:47:38.427 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cache", 2024-03-26T06:47:38.427 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.428 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.428 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.428 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.428 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.428 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.428 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.428 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.428 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.429 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 -- 172.21.15.38:0/1047302382 shutdown_connections 2024-03-26T06:47:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fee340a2970 0x7fee340a4e30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fee4c197a20 0x7fee4c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fee4c106610 0x7fee4c19ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 --2- 172.21.15.38:0/1047302382 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee4c069c10 0x7fee4c19d870 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:47:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 -- 172.21.15.38:0/1047302382 >> 172.21.15.38:0/1047302382 conn(0x7fee4c077290 msgr2=0x7fee4c0fe420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:47:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 -- 172.21.15.38:0/1047302382 shutdown_connections 2024-03-26T06:47:38.430 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:47:37.168+0000 7fee523cd640 1 -- 172.21.15.38:0/1047302382 wait complete. 2024-03-26T06:47:38.432 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:37 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/66972138' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:47:38.432 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:37 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3249150555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:47:38.432 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:37 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/134569495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:47:38.432 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:38 smithi170 ceph-mon[36912]: pgmap v314: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:38.432 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:38 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1047302382' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-26T06:47:38.432 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:37 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/66972138' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:47:38.432 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:37 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3249150555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:47:38.433 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:37 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/134569495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:47:38.433 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:38 smithi121 ceph-mon[36349]: pgmap v314: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:38.433 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:38 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1047302382' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-26T06:47:38.436 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.436 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.436 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.436 INFO:teuthology.orchestra.run.smithi038.stdout: "exclude_perf_counters": { 2024-03-26T06:47:38.436 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "exclude_perf_counters", 2024-03-26T06:47:38.436 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.436 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.437 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.437 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.437 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.437 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.437 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.437 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-26T06:47:38.437 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-26T06:47:38.437 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.438 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.438 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.438 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.438 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.438 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.438 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.438 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.438 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.439 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.439 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.439 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.439 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.439 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.439 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.439 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.439 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.440 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.440 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.440 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.440 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.440 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.440 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.440 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.440 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.441 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.441 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.441 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.441 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.441 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.441 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.441 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.441 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.441 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.442 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.442 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.442 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.442 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.442 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.442 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.442 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.442 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.443 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.443 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.443 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.443 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.443 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.443 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.443 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.443 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.444 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.444 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.444 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.444 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.444 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.444 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.444 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.444 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.445 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.445 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.445 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.445 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.445 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.445 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.445 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.445 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.446 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.446 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.446 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.446 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.446 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.446 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.446 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.446 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools": { 2024-03-26T06:47:38.447 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools", 2024-03-26T06:47:38.447 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.447 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.447 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.447 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.447 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.447 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.447 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.448 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.448 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.448 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.448 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.448 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.448 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-26T06:47:38.448 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-26T06:47:38.448 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.448 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.449 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.449 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "300", 2024-03-26T06:47:38.449 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.449 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.449 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.449 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.449 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.449 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.450 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.450 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_interval": { 2024-03-26T06:47:38.450 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_interval", 2024-03-26T06:47:38.450 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:38.450 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.450 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.450 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15.0", 2024-03-26T06:47:38.451 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.451 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.451 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.451 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.451 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.451 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.451 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.451 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-03-26T06:47:38.452 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-03-26T06:47:38.452 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.452 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.452 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.452 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-03-26T06:47:38.452 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.452 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.452 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.453 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-26T06:47:38.453 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.453 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.453 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.453 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-03-26T06:47:38.453 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-03-26T06:47:38.453 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.453 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.454 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.454 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "9283", 2024-03-26T06:47:38.454 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.454 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.454 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.454 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-26T06:47:38.454 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.454 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.455 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.455 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.455 INFO:teuthology.orchestra.run.smithi038.stdout: "stale_cache_strategy": { 2024-03-26T06:47:38.455 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stale_cache_strategy", 2024-03-26T06:47:38.455 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.455 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.455 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.455 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "log", 2024-03-26T06:47:38.456 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.456 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.456 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.456 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.456 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.456 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.456 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.456 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-03-26T06:47:38.457 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-03-26T06:47:38.457 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.457 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.457 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.457 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "default", 2024-03-26T06:47:38.457 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.457 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.457 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.458 INFO:teuthology.orchestra.run.smithi038.stdout: "default", 2024-03-26T06:47:38.458 INFO:teuthology.orchestra.run.smithi038.stdout: "error" 2024-03-26T06:47:38.458 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.458 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.458 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.458 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.458 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.459 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.459 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-03-26T06:47:38.459 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-03-26T06:47:38.459 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.459 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.459 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.459 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-03-26T06:47:38.459 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-03-26T06:47:38.460 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-03-26T06:47:38.460 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.460 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.460 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.460 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.460 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.460 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.461 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.461 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-03-26T06:47:38.461 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.461 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.461 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.461 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.461 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.461 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.462 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.462 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.462 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.462 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.462 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.462 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.462 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.462 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.462 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.463 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.463 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.463 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.463 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.463 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.463 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.463 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.463 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.464 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.464 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.464 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.464 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.464 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.464 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.464 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.465 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.465 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.465 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.465 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.465 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.465 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.465 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.465 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.466 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.466 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.466 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.466 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.466 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.466 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.466 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.466 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.467 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.467 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.467 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.467 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.467 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.467 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.467 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.467 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.467 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.468 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.468 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.468 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.468 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.468 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.468 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.468 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.469 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.469 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.469 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.469 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.469 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.469 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.469 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.469 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.469 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.470 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.470 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_snap_create": { 2024-03-26T06:47:38.470 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_snap_create", 2024-03-26T06:47:38.470 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.470 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.470 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.470 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.470 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.471 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.471 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.471 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.471 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.471 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.471 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.471 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.471 INFO:teuthology.orchestra.run.smithi038.stdout: "mirror_snapshot_schedule": { 2024-03-26T06:47:38.471 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirror_snapshot_schedule", 2024-03-26T06:47:38.472 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.472 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.472 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.472 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.472 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.472 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.472 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.472 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.472 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.473 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.473 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.473 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.473 INFO:teuthology.orchestra.run.smithi038.stdout: "trash_purge_schedule": { 2024-03-26T06:47:38.473 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "trash_purge_schedule", 2024-03-26T06:47:38.473 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.473 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.473 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.474 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.474 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.474 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.474 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.474 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.474 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.474 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.474 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.475 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.475 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.475 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.475 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.475 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "restful", 2024-03-26T06:47:38.475 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.475 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.475 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.476 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_auth": { 2024-03-26T06:47:38.476 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_auth", 2024-03-26T06:47:38.476 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.476 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.476 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.476 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.476 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.476 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.476 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.477 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.477 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.477 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.477 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.477 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.477 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-03-26T06:47:38.477 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-03-26T06:47:38.477 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.478 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.478 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.478 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.478 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.478 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.478 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.478 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.479 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.479 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.479 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.479 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.479 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.479 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.479 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.479 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.479 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.480 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.480 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.480 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.480 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.480 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.480 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.480 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.480 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.481 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.481 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.481 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.481 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.481 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.481 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.481 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.481 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.481 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.482 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.482 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.482 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.482 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.482 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.482 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.482 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.482 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.483 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.483 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.483 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.483 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.483 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.483 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.483 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.483 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.483 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.484 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.484 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.484 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.484 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.484 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.484 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.484 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.484 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.485 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.486 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-03-26T06:47:38.487 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-03-26T06:47:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.488 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-03-26T06:47:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-03-26T06:47:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.489 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.490 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw", 2024-03-26T06:47:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.491 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.492 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.493 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.494 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.494 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.494 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.494 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.494 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.494 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.494 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.494 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.495 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.496 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.496 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.496 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.496 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.496 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.496 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.496 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.496 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.497 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.497 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.497 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.497 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.497 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.497 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.497 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.497 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.498 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.498 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.498 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.498 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.498 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.498 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.498 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.498 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.499 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.499 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.499 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.499 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.499 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.499 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.499 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.499 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.500 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.500 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.500 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.500 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.500 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.500 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.500 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rook", 2024-03-26T06:47:38.500 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.501 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.501 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.501 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.501 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.501 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.501 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.501 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.501 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.502 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.502 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.502 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.502 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.502 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.502 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.502 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.502 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.503 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.503 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.503 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.503 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.503 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.503 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.503 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.503 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.504 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.504 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.504 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.504 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.504 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.504 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.504 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.504 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.504 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.505 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.505 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.505 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.505 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.505 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.505 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.505 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.505 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.506 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.506 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.506 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.506 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.506 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.506 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.506 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.506 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.507 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.507 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.507 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.507 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.507 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.507 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.507 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.508 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.508 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.508 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.508 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.508 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.508 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.508 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.508 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.508 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.509 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.509 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.509 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.509 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.509 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.509 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.509 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.509 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_class": { 2024-03-26T06:47:38.510 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "storage_class", 2024-03-26T06:47:38.510 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.510 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.510 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.510 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "local", 2024-03-26T06:47:38.510 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.510 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.510 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.511 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-26T06:47:38.511 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.511 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.511 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.511 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.511 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.511 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.511 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.511 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "selftest", 2024-03-26T06:47:38.512 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.512 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.512 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.512 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.512 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.512 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.512 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.512 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.513 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.514 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.514 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.514 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.514 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.514 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.514 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.514 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.514 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.515 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.515 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.515 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.515 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.515 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.515 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.515 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.515 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.515 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.516 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.516 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.516 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.516 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.516 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.516 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.516 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.516 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.517 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.517 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.517 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.517 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.517 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.517 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.517 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.517 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.518 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.518 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.518 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.518 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.518 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.518 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.518 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.518 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.519 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.519 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.519 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.519 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.519 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.519 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.519 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.519 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.520 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.520 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.520 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.520 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.520 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.520 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.520 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.520 INFO:teuthology.orchestra.run.smithi038.stdout: "roption1": { 2024-03-26T06:47:38.521 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption1", 2024-03-26T06:47:38.521 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.521 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.521 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.521 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.521 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.521 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.521 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.521 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.522 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.522 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.522 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.522 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.522 INFO:teuthology.orchestra.run.smithi038.stdout: "roption2": { 2024-03-26T06:47:38.522 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption2", 2024-03-26T06:47:38.522 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.522 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.523 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.523 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "xyz", 2024-03-26T06:47:38.523 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.523 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.523 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.523 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.523 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.523 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.523 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.524 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.524 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption1": { 2024-03-26T06:47:38.524 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption1", 2024-03-26T06:47:38.524 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.524 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.524 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.524 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.524 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.524 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.525 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.525 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.525 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.525 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.525 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.525 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.525 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption2": { 2024-03-26T06:47:38.526 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption2", 2024-03-26T06:47:38.526 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.526 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.526 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.526 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.526 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.526 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.526 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.526 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.527 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.527 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.527 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.527 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.527 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption3": { 2024-03-26T06:47:38.527 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption3", 2024-03-26T06:47:38.527 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:38.527 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.528 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.528 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.528 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.528 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.528 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.528 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.528 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.528 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.529 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.529 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.529 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption4": { 2024-03-26T06:47:38.529 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption4", 2024-03-26T06:47:38.529 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.529 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.529 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.529 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.529 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.530 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.530 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.530 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.530 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.530 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.530 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.530 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.530 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption5": { 2024-03-26T06:47:38.531 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption5", 2024-03-26T06:47:38.531 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.531 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.531 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.531 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.531 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.531 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.531 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.532 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.532 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.532 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.532 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.532 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.532 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption6": { 2024-03-26T06:47:38.532 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption6", 2024-03-26T06:47:38.532 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.532 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.533 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.533 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.533 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.533 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.533 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.533 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.533 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.533 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.533 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.534 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.534 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption7": { 2024-03-26T06:47:38.534 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption7", 2024-03-26T06:47:38.534 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.534 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.534 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.534 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.534 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:38.535 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "42", 2024-03-26T06:47:38.535 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.535 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.535 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.535 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.535 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.535 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.535 INFO:teuthology.orchestra.run.smithi038.stdout: "testkey": { 2024-03-26T06:47:38.535 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testkey", 2024-03-26T06:47:38.536 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.536 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.536 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.536 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.536 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.536 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.536 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.536 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.537 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.537 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.537 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.537 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.537 INFO:teuthology.orchestra.run.smithi038.stdout: "testlkey": { 2024-03-26T06:47:38.537 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testlkey", 2024-03-26T06:47:38.537 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.537 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.538 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.538 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.538 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.538 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.538 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.538 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.538 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.538 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.539 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.539 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.539 INFO:teuthology.orchestra.run.smithi038.stdout: "testnewline": { 2024-03-26T06:47:38.539 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testnewline", 2024-03-26T06:47:38.539 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.539 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.539 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.539 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.540 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.540 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.540 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.540 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.540 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.540 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.540 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.540 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.541 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.541 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.541 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-03-26T06:47:38.541 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.541 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.541 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.541 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_m_granularity": { 2024-03-26T06:47:38.541 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_m_granularity", 2024-03-26T06:47:38.542 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.542 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.542 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.542 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.542 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.542 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.542 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.542 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow minute scheduled snapshots", 2024-03-26T06:47:38.543 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.543 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.543 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.543 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.543 INFO:teuthology.orchestra.run.smithi038.stdout: "dump_on_update": { 2024-03-26T06:47:38.543 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dump_on_update", 2024-03-26T06:47:38.543 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.543 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.543 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.544 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.544 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.544 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.544 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.544 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "dump database to debug log on update", 2024-03-26T06:47:38.544 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.544 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.544 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.545 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.545 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.545 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.545 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.545 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.545 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.545 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.545 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.545 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.546 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.546 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.546 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.546 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.546 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.546 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.546 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.546 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.547 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.547 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.547 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.547 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.547 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.547 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.547 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.547 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.548 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.548 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.548 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.548 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.548 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.548 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.548 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.548 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.548 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.549 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.549 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.549 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.549 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.549 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.549 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.549 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.549 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.549 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.550 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.550 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.550 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.550 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.550 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.550 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.550 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.550 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.551 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.551 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.551 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.551 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.551 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.551 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.551 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.551 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.552 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.552 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.552 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.552 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.552 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.552 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.552 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.553 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.553 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.553 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.553 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.553 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.553 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.553 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.553 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.553 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stats", 2024-03-26T06:47:38.554 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.554 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.554 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.554 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.554 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.554 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.554 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.554 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.555 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.555 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.555 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.555 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.555 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.555 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.555 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.555 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.555 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.556 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.556 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.556 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.556 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.556 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.556 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.556 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.556 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.557 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.557 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.557 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.557 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.557 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.557 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.557 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.557 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.558 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.558 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.558 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.558 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.558 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.558 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.558 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.558 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.559 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.559 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.559 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.559 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.559 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.559 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.559 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.559 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.559 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.560 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.560 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.560 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.560 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.560 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.560 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.560 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.560 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.560 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.561 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.561 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.561 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.561 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.561 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.561 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.561 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.561 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.561 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.562 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.562 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.562 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.562 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.562 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.562 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.562 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.562 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.563 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "status", 2024-03-26T06:47:38.563 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.563 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.563 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.563 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.563 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.563 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.563 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.564 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.564 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.564 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.564 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.564 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.564 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.564 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.564 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.564 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.565 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.565 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.565 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.565 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.565 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.565 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.565 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.565 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.565 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.566 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.566 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.566 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.566 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.566 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.566 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.566 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.567 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.567 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.567 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.567 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.567 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.567 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.567 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.567 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.567 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.568 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.568 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.568 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.568 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.568 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.568 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.568 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.568 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.569 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.569 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.569 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.569 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.569 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.569 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.569 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.569 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.569 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.570 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.570 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.570 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.570 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.570 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.570 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.570 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.570 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.571 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.571 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.571 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.571 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.571 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.571 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.571 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.571 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.572 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.572 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.572 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telegraf", 2024-03-26T06:47:38.572 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.572 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.572 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.572 INFO:teuthology.orchestra.run.smithi038.stdout: "address": { 2024-03-26T06:47:38.572 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "address", 2024-03-26T06:47:38.572 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.573 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.573 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.573 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-26T06:47:38.573 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.573 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.573 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.573 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.573 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.573 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.574 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.574 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.574 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:38.574 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:38.574 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.574 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.574 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.574 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15", 2024-03-26T06:47:38.575 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.575 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.575 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.575 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.575 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.575 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.575 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.575 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.576 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.576 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.576 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.576 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.576 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.576 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.576 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.576 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.576 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.577 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.577 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.577 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.577 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.577 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.577 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.577 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.577 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.578 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.578 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.578 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.578 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.578 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.578 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.578 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.578 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.579 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.579 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.579 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.579 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.579 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.579 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.579 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.579 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.579 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.580 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.580 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.580 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.580 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.580 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.580 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.580 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.580 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.581 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.581 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.581 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.581 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.581 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.581 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.581 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.581 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.581 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.582 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.582 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.582 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.582 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.582 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.582 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.582 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.583 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.583 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.583 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.583 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.583 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.583 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.583 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.583 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.583 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.584 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.584 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.584 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.584 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.584 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.584 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.584 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telemetry", 2024-03-26T06:47:38.584 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.584 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.585 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.585 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_basic": { 2024-03-26T06:47:38.585 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_basic", 2024-03-26T06:47:38.585 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.585 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.585 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.585 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.585 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.586 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.586 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.586 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-26T06:47:38.586 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.586 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.586 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.586 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.586 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_crash": { 2024-03-26T06:47:38.587 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_crash", 2024-03-26T06:47:38.587 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.587 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.587 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.587 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.587 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.587 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.587 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.588 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-26T06:47:38.588 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.588 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.588 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.588 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.588 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_device": { 2024-03-26T06:47:38.588 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_device", 2024-03-26T06:47:38.588 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.589 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.589 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.589 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.589 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.589 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.589 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.589 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-26T06:47:38.589 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.590 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.590 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.590 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.590 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_ident": { 2024-03-26T06:47:38.590 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_ident", 2024-03-26T06:47:38.590 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.590 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.590 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.591 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.591 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.591 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.591 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.591 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-26T06:47:38.591 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.591 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.591 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.591 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.592 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_perf": { 2024-03-26T06:47:38.592 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_perf", 2024-03-26T06:47:38.592 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.592 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.592 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.592 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.592 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.592 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.592 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.593 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-26T06:47:38.593 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.593 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.593 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.593 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.593 INFO:teuthology.orchestra.run.smithi038.stdout: "contact": { 2024-03-26T06:47:38.593 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "contact", 2024-03-26T06:47:38.593 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.593 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.594 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.594 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.594 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.594 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.594 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.594 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.594 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.594 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.595 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.595 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.595 INFO:teuthology.orchestra.run.smithi038.stdout: "description": { 2024-03-26T06:47:38.595 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "description", 2024-03-26T06:47:38.595 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.595 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.595 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.595 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.596 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.596 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.596 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.596 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.596 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.596 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.596 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.596 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.596 INFO:teuthology.orchestra.run.smithi038.stdout: "device_url": { 2024-03-26T06:47:38.597 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_url", 2024-03-26T06:47:38.597 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.597 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.597 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.597 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-26T06:47:38.597 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.597 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.597 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.598 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.598 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.598 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.598 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.598 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.598 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-03-26T06:47:38.598 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-03-26T06:47:38.599 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.599 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.599 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.599 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.599 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.599 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.599 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.599 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.599 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.600 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.600 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.600 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.600 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:38.600 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:38.600 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.600 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.600 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.601 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "24", 2024-03-26T06:47:38.601 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "8", 2024-03-26T06:47:38.601 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.601 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.601 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.601 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.601 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.601 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.601 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.602 INFO:teuthology.orchestra.run.smithi038.stdout: "last_opt_revision": { 2024-03-26T06:47:38.602 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "last_opt_revision", 2024-03-26T06:47:38.602 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.602 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.602 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.602 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1", 2024-03-26T06:47:38.602 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.602 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.603 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.603 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.603 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.603 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.603 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.603 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.603 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard": { 2024-03-26T06:47:38.603 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard", 2024-03-26T06:47:38.604 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.604 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.604 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.604 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.604 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.604 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.604 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.604 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.604 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.605 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.605 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.605 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.605 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard_description": { 2024-03-26T06:47:38.605 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard_description", 2024-03-26T06:47:38.605 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.605 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.605 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.606 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.606 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.606 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.606 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.606 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.606 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.606 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.606 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.606 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.607 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.607 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.607 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.607 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.607 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.607 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.607 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.607 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.608 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.608 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.608 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.608 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.608 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.608 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.608 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.608 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.609 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.609 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.609 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.609 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.609 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.609 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.609 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.609 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.610 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.610 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.610 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.610 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.610 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.610 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.610 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.610 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.611 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.611 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.611 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.611 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.611 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.611 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.611 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.611 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.611 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.612 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.612 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.612 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.612 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.612 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.612 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.612 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.612 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.613 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.613 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.613 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.613 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.613 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.613 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.613 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.613 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.613 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.614 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.614 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.614 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.614 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.614 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.614 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.614 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.614 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.615 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.615 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.615 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.615 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.615 INFO:teuthology.orchestra.run.smithi038.stdout: "organization": { 2024-03-26T06:47:38.615 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "organization", 2024-03-26T06:47:38.615 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.615 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.616 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.616 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.616 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.616 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.616 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.616 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.616 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.616 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.616 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.617 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.617 INFO:teuthology.orchestra.run.smithi038.stdout: "proxy": { 2024-03-26T06:47:38.617 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "proxy", 2024-03-26T06:47:38.617 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.617 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.617 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.617 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.617 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.618 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.618 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.618 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.618 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.618 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.618 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.618 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.618 INFO:teuthology.orchestra.run.smithi038.stdout: "url": { 2024-03-26T06:47:38.619 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url", 2024-03-26T06:47:38.619 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.619 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.619 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.619 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-26T06:47:38.619 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.619 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.619 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.621 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.622 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.622 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.622 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.622 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.622 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.622 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.622 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.623 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test_orchestrator", 2024-03-26T06:47:38.623 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.623 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.623 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.623 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.623 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.623 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.623 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.623 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.624 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.624 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.624 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.624 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.624 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.624 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.624 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.624 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.625 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.625 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.625 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.625 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.625 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.625 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.625 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.625 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.625 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.626 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.626 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.626 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.626 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.626 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.626 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.626 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.626 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.627 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.627 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.627 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.627 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.627 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.627 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.627 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.627 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.628 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.628 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.628 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.628 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.628 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.628 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.628 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.628 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.628 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.629 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.629 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.629 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.629 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.629 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.629 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.629 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.629 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.630 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.630 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.630 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.630 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.630 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.630 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.630 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.630 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.631 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.631 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.631 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.631 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.631 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.632 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.632 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-03-26T06:47:38.632 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.632 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.632 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.632 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.632 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.632 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.632 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.633 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.633 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.633 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.633 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.633 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.633 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.633 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.633 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.634 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.634 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.634 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.634 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.634 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.634 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.634 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.634 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.634 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.635 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.635 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.635 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.635 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.635 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.635 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.635 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.635 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.636 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.636 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.636 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.636 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.636 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.636 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.636 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.636 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.636 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.637 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.637 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.637 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.637 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.637 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.637 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.637 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.637 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.638 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.638 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.638 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.638 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.638 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.638 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.638 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.638 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.639 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.639 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.639 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.639 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.639 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.639 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.639 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.639 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.640 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.640 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.640 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.640 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.640 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.640 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.640 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.640 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.640 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_clones": { 2024-03-26T06:47:38.641 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_clones", 2024-03-26T06:47:38.641 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.641 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.641 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.641 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4", 2024-03-26T06:47:38.641 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.641 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.641 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.642 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-26T06:47:38.642 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.642 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.642 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.642 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.642 INFO:teuthology.orchestra.run.smithi038.stdout: "periodic_async_work": { 2024-03-26T06:47:38.642 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "periodic_async_work", 2024-03-26T06:47:38.642 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.643 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.643 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.643 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.643 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.643 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.643 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.643 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Periodically check for async work", 2024-03-26T06:47:38.643 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.643 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.644 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.644 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.644 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_delay": { 2024-03-26T06:47:38.644 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_delay", 2024-03-26T06:47:38.644 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.644 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.644 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.644 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:38.644 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.645 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.645 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.645 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-26T06:47:38.645 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.645 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.645 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.645 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.645 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_no_wait": { 2024-03-26T06:47:38.646 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_no_wait", 2024-03-26T06:47:38.646 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.646 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.646 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.646 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.646 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.646 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.646 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.647 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-26T06:47:38.647 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.647 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.647 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.647 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.647 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.647 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.647 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.648 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix", 2024-03-26T06:47:38.648 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.648 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.648 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.648 INFO:teuthology.orchestra.run.smithi038.stdout: "discovery_interval": { 2024-03-26T06:47:38.648 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "discovery_interval", 2024-03-26T06:47:38.648 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:38.648 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.648 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.649 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "100", 2024-03-26T06:47:38.649 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.649 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.649 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.649 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.649 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.649 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.649 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.650 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.650 INFO:teuthology.orchestra.run.smithi038.stdout: "identifier": { 2024-03-26T06:47:38.650 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "identifier", 2024-03-26T06:47:38.650 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.650 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.650 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.650 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.650 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.651 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.651 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.651 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.651 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.651 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.651 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.651 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.651 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:38.652 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:38.652 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.652 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.652 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.652 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:38.652 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.652 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.652 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.653 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.653 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.653 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.653 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.653 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.653 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.653 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.653 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.653 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.654 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.654 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.654 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.654 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.654 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.654 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.654 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.654 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.655 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.655 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.655 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.655 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.655 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.655 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.655 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.655 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.656 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.656 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.656 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.656 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.656 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.656 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.656 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.656 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.656 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.657 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.657 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.657 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.657 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.657 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.657 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.657 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.657 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.657 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.658 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.658 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.658 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.658 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.658 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.658 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.658 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.659 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.659 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.659 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.659 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.659 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.659 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.659 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.659 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.659 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.660 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.660 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.660 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.660 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.660 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.660 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.660 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.660 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.660 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.661 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.661 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.661 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.661 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.661 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.661 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.661 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.661 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_host": { 2024-03-26T06:47:38.661 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_host", 2024-03-26T06:47:38.662 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.662 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.662 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.662 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.662 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.662 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.662 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.662 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.663 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.663 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.663 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.663 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.663 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_port": { 2024-03-26T06:47:38.663 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_port", 2024-03-26T06:47:38.663 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.663 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.663 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.664 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10051", 2024-03-26T06:47:38.664 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.664 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.664 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.664 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.664 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.664 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.664 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.664 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.665 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_sender": { 2024-03-26T06:47:38.665 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_sender", 2024-03-26T06:47:38.665 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.665 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.665 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.665 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-26T06:47:38.665 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.665 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.665 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.666 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.666 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.666 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.666 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.666 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.666 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.666 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.666 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:47:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: "modules": [ 2024-03-26T06:47:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-03-26T06:47:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: "dashboard", 2024-03-26T06:47:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: "iostat", 2024-03-26T06:47:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: "nfs", 2024-03-26T06:47:38.667 INFO:teuthology.orchestra.run.smithi038.stdout: "restful", 2024-03-26T06:47:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "snap_schedule" 2024-03-26T06:47:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "available_modules": [ 2024-03-26T06:47:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "alerts", 2024-03-26T06:47:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.668 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.669 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How frequently to reexamine health status", 2024-03-26T06:47:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.670 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.671 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.672 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.673 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.674 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.675 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.676 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.676 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.676 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.676 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.676 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.676 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.676 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.676 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.677 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.677 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.677 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.677 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.677 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.677 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.677 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.677 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.677 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.678 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.678 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.678 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.678 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.678 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.678 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.678 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.678 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_destination": { 2024-03-26T06:47:38.679 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_destination", 2024-03-26T06:47:38.679 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.679 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.679 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.679 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.679 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.679 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.679 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.680 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email address to send alerts to", 2024-03-26T06:47:38.680 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.680 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.680 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.680 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.680 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_from_name": { 2024-03-26T06:47:38.680 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_from_name", 2024-03-26T06:47:38.680 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.680 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.681 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.681 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "Ceph", 2024-03-26T06:47:38.681 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.681 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.681 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.681 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Email From: name", 2024-03-26T06:47:38.681 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.681 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.681 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.682 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.682 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_host": { 2024-03-26T06:47:38.682 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_host", 2024-03-26T06:47:38.682 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.682 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.682 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.682 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.682 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.683 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.683 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.683 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP server", 2024-03-26T06:47:38.683 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.683 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.683 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.683 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.683 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_password": { 2024-03-26T06:47:38.683 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_password", 2024-03-26T06:47:38.684 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.684 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.684 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.684 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.684 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.684 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.684 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.684 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Password to authenticate with", 2024-03-26T06:47:38.684 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.685 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.685 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.685 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.685 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_port": { 2024-03-26T06:47:38.685 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_port", 2024-03-26T06:47:38.685 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.685 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.685 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.685 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "465", 2024-03-26T06:47:38.686 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.686 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.686 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.686 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP port", 2024-03-26T06:47:38.686 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.686 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.686 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.686 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.687 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_sender": { 2024-03-26T06:47:38.687 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_sender", 2024-03-26T06:47:38.687 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.687 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.687 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.687 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.687 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.687 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SMTP envelope sender", 2024-03-26T06:47:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_ssl": { 2024-03-26T06:47:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_ssl", 2024-03-26T06:47:38.688 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.689 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.689 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.689 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.689 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.689 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.689 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.689 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-26T06:47:38.689 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.690 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.690 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.690 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.690 INFO:teuthology.orchestra.run.smithi038.stdout: "smtp_user": { 2024-03-26T06:47:38.690 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "smtp_user", 2024-03-26T06:47:38.690 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.690 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.690 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.690 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.691 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.691 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.691 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.691 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "User to authenticate as", 2024-03-26T06:47:38.691 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.691 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.691 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.691 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.692 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.692 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.692 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.692 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "balancer", 2024-03-26T06:47:38.692 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.692 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.692 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.692 INFO:teuthology.orchestra.run.smithi038.stdout: "active": { 2024-03-26T06:47:38.693 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "active", 2024-03-26T06:47:38.693 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.693 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.693 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.693 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.693 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.693 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.693 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.693 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically balance PGs across cluster", 2024-03-26T06:47:38.694 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.694 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.694 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.694 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.694 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_time": { 2024-03-26T06:47:38.694 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_time", 2024-03-26T06:47:38.694 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.694 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.695 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.695 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0000", 2024-03-26T06:47:38.695 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.695 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.695 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.695 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "beginning time of day to automatically balance", 2024-03-26T06:47:38.695 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-26T06:47:38.695 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.695 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.696 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.696 INFO:teuthology.orchestra.run.smithi038.stdout: "begin_weekday": { 2024-03-26T06:47:38.696 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "begin_weekday", 2024-03-26T06:47:38.696 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:38.696 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.696 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.696 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:38.696 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-03-26T06:47:38.697 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-03-26T06:47:38.697 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.697 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-26T06:47:38.697 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-26T06:47:38.697 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.697 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.697 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.697 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_max_iterations": { 2024-03-26T06:47:38.698 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_max_iterations", 2024-03-26T06:47:38.698 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:38.698 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.698 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.698 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "25", 2024-03-26T06:47:38.698 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:38.698 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "250", 2024-03-26T06:47:38.698 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.699 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-26T06:47:38.699 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.699 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.699 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.699 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.699 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_metrics": { 2024-03-26T06:47:38.699 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_metrics", 2024-03-26T06:47:38.699 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.700 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.700 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.700 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "pgs,objects,bytes", 2024-03-26T06:47:38.700 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.700 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.700 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.700 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-26T06:47:38.700 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-26T06:47:38.700 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.701 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.701 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.701 INFO:teuthology.orchestra.run.smithi038.stdout: "crush_compat_step": { 2024-03-26T06:47:38.701 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crush_compat_step", 2024-03-26T06:47:38.701 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:38.701 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.701 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.701 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.5", 2024-03-26T06:47:38.702 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0.001", 2024-03-26T06:47:38.702 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "0.999", 2024-03-26T06:47:38.702 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.702 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "aggressiveness of optimization", 2024-03-26T06:47:38.702 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-26T06:47:38.702 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.702 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.702 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.703 INFO:teuthology.orchestra.run.smithi038.stdout: "end_time": { 2024-03-26T06:47:38.703 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_time", 2024-03-26T06:47:38.703 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.703 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.703 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.703 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2359", 2024-03-26T06:47:38.703 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.703 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.703 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.704 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ending time of day to automatically balance", 2024-03-26T06:47:38.704 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-26T06:47:38.704 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.704 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.704 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.704 INFO:teuthology.orchestra.run.smithi038.stdout: "end_weekday": { 2024-03-26T06:47:38.704 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "end_weekday", 2024-03-26T06:47:38.704 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:38.704 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.705 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.705 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:38.705 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "0", 2024-03-26T06:47:38.705 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "6", 2024-03-26T06:47:38.705 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.705 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-26T06:47:38.705 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-26T06:47:38.705 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.706 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.706 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.706 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.706 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.706 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.706 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.706 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.706 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.707 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.707 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.707 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.707 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.707 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.707 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.707 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.707 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.708 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.708 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.708 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.708 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.708 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.708 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.708 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.708 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.709 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.709 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.709 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.709 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.709 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.709 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.709 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.709 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.709 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.710 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.710 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.710 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.710 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.710 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.710 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.710 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.710 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.711 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.711 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.711 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.711 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.711 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.711 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.711 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.711 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.711 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.712 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.712 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.712 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.712 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.712 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.712 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.712 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.712 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.712 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.713 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.713 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.713 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.713 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.713 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.713 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.713 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.713 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: "min_score": { 2024-03-26T06:47:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_score", 2024-03-26T06:47:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:38.714 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-26T06:47:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.715 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-03-26T06:47:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-03-26T06:47:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.716 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "upmap", 2024-03-26T06:47:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "crush-compat", 2024-03-26T06:47:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "none", 2024-03-26T06:47:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "read", 2024-03-26T06:47:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap", 2024-03-26T06:47:38.717 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap-read" 2024-03-26T06:47:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Balancer mode", 2024-03-26T06:47:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_ids": { 2024-03-26T06:47:38.718 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_ids", 2024-03-26T06:47:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-26T06:47:38.719 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.720 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-26T06:47:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.721 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_deviation": { 2024-03-26T06:47:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_deviation", 2024-03-26T06:47:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:38.722 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-26T06:47:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-26T06:47:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: "upmap_max_optimizations": { 2024-03-26T06:47:38.723 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "upmap_max_optimizations", 2024-03-26T06:47:38.724 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:38.724 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.724 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.724 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.724 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.724 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.724 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.724 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-26T06:47:38.725 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.725 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.725 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.725 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.725 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.725 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.725 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.725 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm", 2024-03-26T06:47:38.725 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.726 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.726 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.726 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_down_multiplier": { 2024-03-26T06:47:38.726 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_down_multiplier", 2024-03-26T06:47:38.726 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:38.726 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.726 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.726 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-03-26T06:47:38.727 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.727 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.727 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.727 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-26T06:47:38.727 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.727 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.727 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.727 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.727 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_refresh_rate": { 2024-03-26T06:47:38.728 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_refresh_rate", 2024-03-26T06:47:38.728 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.728 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.728 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.728 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "20", 2024-03-26T06:47:38.728 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.728 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.728 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.729 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-26T06:47:38.729 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.729 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.729 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.729 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.729 INFO:teuthology.orchestra.run.smithi038.stdout: "agent_starting_port": { 2024-03-26T06:47:38.729 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "agent_starting_port", 2024-03-26T06:47:38.729 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.729 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.730 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.730 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4721", 2024-03-26T06:47:38.730 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.730 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.730 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.730 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-26T06:47:38.730 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.730 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_ptrace": { 2024-03-26T06:47:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_ptrace", 2024-03-26T06:47:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.731 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-26T06:47:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-26T06:47:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.732 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_interval": { 2024-03-26T06:47:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_interval", 2024-03-26T06:47:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.733 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-26T06:47:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "autotune_memory_target_ratio": { 2024-03-26T06:47:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "autotune_memory_target_ratio", 2024-03-26T06:47:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.734 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0.7", 2024-03-26T06:47:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-26T06:47:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.735 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm_log_destination": { 2024-03-26T06:47:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cephadm_log_destination", 2024-03-26T06:47:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.736 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "file", 2024-03-26T06:47:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "file,syslog", 2024-03-26T06:47:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "syslog" 2024-03-26T06:47:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-26T06:47:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.737 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: "cgroups_split": { 2024-03-26T06:47:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cgroups_split", 2024-03-26T06:47:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.738 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-26T06:47:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.739 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.740 INFO:teuthology.orchestra.run.smithi038.stdout: "config_checks_enabled": { 2024-03-26T06:47:38.740 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_checks_enabled", 2024-03-26T06:47:38.740 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.740 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.740 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.740 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.740 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.740 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.740 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.741 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-26T06:47:38.741 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.741 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.741 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.741 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.741 INFO:teuthology.orchestra.run.smithi038.stdout: "config_dashboard": { 2024-03-26T06:47:38.741 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_dashboard", 2024-03-26T06:47:38.741 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.741 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.742 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.742 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.742 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.742 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.742 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.742 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-26T06:47:38.742 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.742 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.743 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.743 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.743 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_alertmanager": { 2024-03-26T06:47:38.743 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_alertmanager", 2024-03-26T06:47:38.743 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.743 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.743 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.743 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-26T06:47:38.743 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.744 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.744 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.744 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:38.744 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.744 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.744 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.744 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.744 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_base": { 2024-03-26T06:47:38.744 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_base", 2024-03-26T06:47:38.745 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.745 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.745 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.745 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-26T06:47:38.745 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.745 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.745 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.745 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Container image name, without the tag", 2024-03-26T06:47:38.746 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.746 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.746 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.746 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.746 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_elasticsearch": { 2024-03-26T06:47:38.746 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_elasticsearch", 2024-03-26T06:47:38.746 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.746 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.747 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.747 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-26T06:47:38.747 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.747 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.747 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.747 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "elasticsearch container image", 2024-03-26T06:47:38.747 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.747 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.748 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.748 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.748 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_grafana": { 2024-03-26T06:47:38.748 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_grafana", 2024-03-26T06:47:38.748 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.748 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.748 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.748 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-26T06:47:38.748 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.749 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.749 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.749 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:38.749 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.749 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.749 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.749 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.749 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_haproxy": { 2024-03-26T06:47:38.750 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_haproxy", 2024-03-26T06:47:38.750 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.750 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.750 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.750 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-26T06:47:38.750 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.750 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.750 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.750 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "HAproxy container image", 2024-03-26T06:47:38.751 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.751 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.751 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.751 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.751 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_agent": { 2024-03-26T06:47:38.751 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_agent", 2024-03-26T06:47:38.751 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.751 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.752 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.752 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-26T06:47:38.752 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.752 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.752 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.752 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger agent container image", 2024-03-26T06:47:38.752 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.752 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.753 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.753 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.753 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_collector": { 2024-03-26T06:47:38.753 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_collector", 2024-03-26T06:47:38.753 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.753 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.753 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.753 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-26T06:47:38.753 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.754 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.754 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.754 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger collector container image", 2024-03-26T06:47:38.754 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.754 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.754 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.754 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.754 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_jaeger_query": { 2024-03-26T06:47:38.755 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_jaeger_query", 2024-03-26T06:47:38.755 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.755 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.755 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.755 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-26T06:47:38.755 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.755 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.755 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.756 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Jaeger query container image", 2024-03-26T06:47:38.756 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.756 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.756 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.756 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.756 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_keepalived": { 2024-03-26T06:47:38.756 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_keepalived", 2024-03-26T06:47:38.756 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.756 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.757 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.757 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-26T06:47:38.757 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.757 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.757 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.757 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Keepalived container image", 2024-03-26T06:47:38.757 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.757 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.758 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.758 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.758 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_loki": { 2024-03-26T06:47:38.758 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_loki", 2024-03-26T06:47:38.758 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.758 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.758 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.758 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-26T06:47:38.759 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.759 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.759 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.759 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Loki container image", 2024-03-26T06:47:38.759 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.759 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.759 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.759 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.759 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_node_exporter": { 2024-03-26T06:47:38.760 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_node_exporter", 2024-03-26T06:47:38.760 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.760 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.760 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.760 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-26T06:47:38.760 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.760 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.760 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.761 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:38.761 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.761 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.761 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.761 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.761 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_nvmeof": { 2024-03-26T06:47:38.761 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_nvmeof", 2024-03-26T06:47:38.761 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.761 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.762 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.762 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-26T06:47:38.762 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.762 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.762 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.762 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Nvme-of container image", 2024-03-26T06:47:38.762 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.762 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.763 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.763 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.763 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_prometheus": { 2024-03-26T06:47:38.763 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_prometheus", 2024-03-26T06:47:38.763 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.763 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.763 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.763 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-26T06:47:38.764 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.764 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.764 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.764 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Prometheus container image", 2024-03-26T06:47:38.764 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.764 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.764 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.764 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.764 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_promtail": { 2024-03-26T06:47:38.765 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_promtail", 2024-03-26T06:47:38.765 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.765 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.765 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.765 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-26T06:47:38.765 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.765 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.765 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.766 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Promtail container image", 2024-03-26T06:47:38.766 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.766 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.766 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.766 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.766 INFO:teuthology.orchestra.run.smithi038.stdout: "container_image_snmp_gateway": { 2024-03-26T06:47:38.766 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_image_snmp_gateway", 2024-03-26T06:47:38.766 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.767 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.767 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.767 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-26T06:47:38.767 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.767 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.767 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.767 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "SNMP Gateway container image", 2024-03-26T06:47:38.768 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.768 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.768 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.768 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.768 INFO:teuthology.orchestra.run.smithi038.stdout: "container_init": { 2024-03-26T06:47:38.768 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "container_init", 2024-03-26T06:47:38.768 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.768 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.769 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.769 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.769 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.769 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.769 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.769 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Run podman/docker with `--init`", 2024-03-26T06:47:38.769 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.769 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.770 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.770 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.770 INFO:teuthology.orchestra.run.smithi038.stdout: "daemon_cache_timeout": { 2024-03-26T06:47:38.770 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "daemon_cache_timeout", 2024-03-26T06:47:38.770 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.770 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.770 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.770 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:38.771 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.771 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.771 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.771 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-26T06:47:38.771 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.771 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.771 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.771 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.772 INFO:teuthology.orchestra.run.smithi038.stdout: "default_cephadm_command_timeout": { 2024-03-26T06:47:38.772 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_cephadm_command_timeout", 2024-03-26T06:47:38.772 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.772 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.772 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.772 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "900", 2024-03-26T06:47:38.772 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.772 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.772 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.773 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-26T06:47:38.773 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.773 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.773 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.773 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.773 INFO:teuthology.orchestra.run.smithi038.stdout: "default_registry": { 2024-03-26T06:47:38.773 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "default_registry", 2024-03-26T06:47:38.773 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.774 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.774 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.774 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "docker.io", 2024-03-26T06:47:38.774 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.774 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.774 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.774 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-26T06:47:38.774 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.774 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.775 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.775 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.775 INFO:teuthology.orchestra.run.smithi038.stdout: "device_cache_timeout": { 2024-03-26T06:47:38.775 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_cache_timeout", 2024-03-26T06:47:38.775 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.775 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.775 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.775 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1800", 2024-03-26T06:47:38.776 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.776 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.776 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.776 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache device inventory", 2024-03-26T06:47:38.776 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.776 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.776 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.776 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.776 INFO:teuthology.orchestra.run.smithi038.stdout: "device_enhanced_scan": { 2024-03-26T06:47:38.777 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_enhanced_scan", 2024-03-26T06:47:38.777 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.777 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.777 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.777 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.777 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.777 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.777 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.778 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-26T06:47:38.778 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.778 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.778 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.778 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.778 INFO:teuthology.orchestra.run.smithi038.stdout: "facts_cache_timeout": { 2024-03-26T06:47:38.778 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "facts_cache_timeout", 2024-03-26T06:47:38.778 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.779 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.779 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.779 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:38.779 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.779 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.779 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.779 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "seconds to cache host facts data", 2024-03-26T06:47:38.779 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.780 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.780 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.780 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.780 INFO:teuthology.orchestra.run.smithi038.stdout: "host_check_interval": { 2024-03-26T06:47:38.780 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "host_check_interval", 2024-03-26T06:47:38.780 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.780 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.780 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.780 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:38.781 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.781 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.781 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.781 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to perform a host check", 2024-03-26T06:47:38.781 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.781 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.781 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.781 INFO:teuthology.orchestra.run.smithi038.stdout: "hw_monitoring": { 2024-03-26T06:47:38.782 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hw_monitoring", 2024-03-26T06:47:38.782 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.782 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.782 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.782 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.782 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.782 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.782 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.783 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-26T06:47:38.783 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.783 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.783 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.783 INFO:teuthology.orchestra.run.smithi038.stdout: "inventory_list_all": { 2024-03-26T06:47:38.783 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "inventory_list_all", 2024-03-26T06:47:38.783 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.783 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.784 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.784 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.784 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.784 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.784 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.784 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-26T06:47:38.784 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.784 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.784 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.785 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.785 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.785 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.785 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.785 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.785 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.785 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.786 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.786 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.786 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.786 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.786 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.786 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.786 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.786 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.787 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.787 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.787 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.787 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.787 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.787 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.787 INFO:teuthology.orchestra.run.smithi038.stdout: "log_refresh_metadata": { 2024-03-26T06:47:38.787 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_refresh_metadata", 2024-03-26T06:47:38.788 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.788 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.788 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.788 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.788 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.788 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.788 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.788 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-26T06:47:38.788 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.789 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.789 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.789 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.789 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.789 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.789 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.789 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.789 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.790 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.790 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.790 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.790 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-26T06:47:38.790 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.790 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.790 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.790 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.791 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.791 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.791 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.791 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.791 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.791 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.791 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.791 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.791 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.792 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.792 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.792 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.792 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.792 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.792 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.792 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.792 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.793 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.793 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.793 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.793 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.793 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.793 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.793 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.793 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.793 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.794 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.794 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.794 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.794 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.794 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.794 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.794 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.794 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.795 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.795 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-26T06:47:38.795 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-26T06:47:38.795 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.795 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.795 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.795 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.795 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.795 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.796 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.796 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-26T06:47:38.796 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.796 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.796 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.796 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.796 INFO:teuthology.orchestra.run.smithi038.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-26T06:47:38.796 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-26T06:47:38.797 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.797 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.797 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.797 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "*", 2024-03-26T06:47:38.797 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.797 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.797 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.797 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-26T06:47:38.798 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.798 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.798 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.798 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.798 INFO:teuthology.orchestra.run.smithi038.stdout: "max_count_per_host": { 2024-03-26T06:47:38.798 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_count_per_host", 2024-03-26T06:47:38.798 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.798 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.799 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.799 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.799 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.799 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.799 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.799 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of daemons per service per host", 2024-03-26T06:47:38.799 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.799 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.799 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.800 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.800 INFO:teuthology.orchestra.run.smithi038.stdout: "max_osd_draining_count": { 2024-03-26T06:47:38.800 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_osd_draining_count", 2024-03-26T06:47:38.800 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.800 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.800 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.800 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.800 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.801 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.801 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.801 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-26T06:47:38.801 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.801 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.801 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.801 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.801 INFO:teuthology.orchestra.run.smithi038.stdout: "migration_current": { 2024-03-26T06:47:38.801 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "migration_current", 2024-03-26T06:47:38.802 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.802 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.802 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.802 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.802 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.802 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.802 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.802 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "internal - do not modify", 2024-03-26T06:47:38.803 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.803 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.803 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.803 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.803 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": { 2024-03-26T06:47:38.803 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mode", 2024-03-26T06:47:38.803 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.803 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.804 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.804 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "root", 2024-03-26T06:47:38.804 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.804 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.804 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.804 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm-package", 2024-03-26T06:47:38.804 INFO:teuthology.orchestra.run.smithi038.stdout: "root" 2024-03-26T06:47:38.804 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.804 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "mode for remote execution of cephadm", 2024-03-26T06:47:38.805 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.805 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.805 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.805 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.805 INFO:teuthology.orchestra.run.smithi038.stdout: "oob_default_addr": { 2024-03-26T06:47:38.805 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "oob_default_addr", 2024-03-26T06:47:38.805 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.805 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.806 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.806 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "169.254.1.1", 2024-03-26T06:47:38.806 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.806 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.806 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.806 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-26T06:47:38.806 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.806 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.806 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.807 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.807 INFO:teuthology.orchestra.run.smithi038.stdout: "prometheus_alerts_path": { 2024-03-26T06:47:38.807 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus_alerts_path", 2024-03-26T06:47:38.807 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.807 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.807 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.807 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-26T06:47:38.807 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.808 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.808 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.808 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-26T06:47:38.808 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.808 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.808 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.808 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.808 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_insecure": { 2024-03-26T06:47:38.809 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_insecure", 2024-03-26T06:47:38.809 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.809 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.809 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.809 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.809 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.809 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.809 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.809 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-26T06:47:38.810 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.810 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.810 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.810 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.810 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_password": { 2024-03-26T06:47:38.810 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_password", 2024-03-26T06:47:38.810 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.810 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.811 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.811 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.811 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.811 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.811 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.811 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-26T06:47:38.811 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.811 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.811 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.812 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.812 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_url": { 2024-03-26T06:47:38.812 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_url", 2024-03-26T06:47:38.812 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.812 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.812 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.812 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.812 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.813 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.813 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.813 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-26T06:47:38.813 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.813 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.813 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.813 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.813 INFO:teuthology.orchestra.run.smithi038.stdout: "registry_username": { 2024-03-26T06:47:38.813 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "registry_username", 2024-03-26T06:47:38.814 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.814 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.814 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.814 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.814 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.814 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.814 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.814 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-26T06:47:38.815 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.815 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.815 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.815 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.815 INFO:teuthology.orchestra.run.smithi038.stdout: "secure_monitoring_stack": { 2024-03-26T06:47:38.815 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "secure_monitoring_stack", 2024-03-26T06:47:38.815 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.815 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.815 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.816 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.816 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.816 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.816 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.816 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-26T06:47:38.816 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.816 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.816 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.817 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.817 INFO:teuthology.orchestra.run.smithi038.stdout: "service_discovery_port": { 2024-03-26T06:47:38.817 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "service_discovery_port", 2024-03-26T06:47:38.817 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.817 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.817 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.817 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8765", 2024-03-26T06:47:38.817 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.818 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.818 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.818 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "cephadm service discovery port", 2024-03-26T06:47:38.818 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.818 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.818 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.818 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.818 INFO:teuthology.orchestra.run.smithi038.stdout: "ssh_config_file": { 2024-03-26T06:47:38.818 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssh_config_file", 2024-03-26T06:47:38.819 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.819 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.819 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.819 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.819 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.819 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.819 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.819 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-26T06:47:38.820 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.820 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.820 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.820 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.820 INFO:teuthology.orchestra.run.smithi038.stdout: "use_agent": { 2024-03-26T06:47:38.820 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_agent", 2024-03-26T06:47:38.820 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.820 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.820 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.821 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.821 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.821 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.821 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.821 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-26T06:47:38.821 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.821 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.821 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.822 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.822 INFO:teuthology.orchestra.run.smithi038.stdout: "use_repo_digest": { 2024-03-26T06:47:38.822 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "use_repo_digest", 2024-03-26T06:47:38.822 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.822 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.822 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.822 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.822 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.823 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.823 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.823 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-26T06:47:38.823 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.823 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.823 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.823 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.823 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_failed_host_check": { 2024-03-26T06:47:38.823 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_failed_host_check", 2024-03-26T06:47:38.824 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.824 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.824 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.824 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.824 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.824 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.824 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.824 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if the host check fails", 2024-03-26T06:47:38.824 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.825 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.825 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.825 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.825 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_daemons": { 2024-03-26T06:47:38.825 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_daemons", 2024-03-26T06:47:38.825 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.825 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.825 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.826 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.826 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.826 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.826 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.826 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-26T06:47:38.826 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.826 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.826 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.826 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.827 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_on_stray_hosts": { 2024-03-26T06:47:38.827 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_on_stray_hosts", 2024-03-26T06:47:38.827 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.827 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.827 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.827 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.827 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.827 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.828 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.828 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-26T06:47:38.828 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.828 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.828 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.828 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.828 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.829 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.829 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crash", 2024-03-26T06:47:38.829 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.829 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.829 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.829 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.829 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.829 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.829 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.830 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.830 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.830 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.830 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.830 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.830 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.830 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.830 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.831 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.831 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.831 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.831 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.831 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.831 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.831 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.831 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.831 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.832 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.832 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.832 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.832 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.832 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.832 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.832 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.832 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.833 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.833 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.833 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.833 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.833 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.833 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.833 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.833 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.833 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.834 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.834 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.834 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.834 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.834 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.834 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.834 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.834 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.835 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.835 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.835 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.835 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.835 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.835 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.835 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.835 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.835 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.836 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.836 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.836 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.836 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.836 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.836 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.836 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.836 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.836 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.837 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.837 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.837 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.837 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.837 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.837 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.837 INFO:teuthology.orchestra.run.smithi038.stdout: "retain_interval": { 2024-03-26T06:47:38.837 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retain_interval", 2024-03-26T06:47:38.838 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.838 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.838 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.838 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "31536000", 2024-03-26T06:47:38.838 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.838 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.838 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.838 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-26T06:47:38.839 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.839 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.839 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.839 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.839 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_recent_interval": { 2024-03-26T06:47:38.839 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_recent_interval", 2024-03-26T06:47:38.839 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.839 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.840 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.840 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1209600", 2024-03-26T06:47:38.840 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.840 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.840 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.840 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-26T06:47:38.840 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.840 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.840 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.841 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.841 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.841 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.841 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dashboard", 2024-03-26T06:47:38.841 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.841 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.841 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.842 INFO:teuthology.orchestra.run.smithi038.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-26T06:47:38.842 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-26T06:47:38.842 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.842 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.842 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.842 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.842 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.842 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.842 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.843 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.843 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.843 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.843 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.843 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.843 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_HOST": { 2024-03-26T06:47:38.843 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-26T06:47:38.843 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.844 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.844 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.844 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.844 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.844 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.844 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.844 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.844 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.844 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.845 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.845 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.845 INFO:teuthology.orchestra.run.smithi038.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-26T06:47:38.845 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-26T06:47:38.845 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.845 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.845 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.845 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.845 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.846 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.846 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.846 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.846 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.846 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.846 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.846 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.846 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_ENABLED": { 2024-03-26T06:47:38.847 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_ENABLED", 2024-03-26T06:47:38.847 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.847 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.847 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.847 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.847 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.847 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.847 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.848 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.848 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.848 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.848 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.848 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.848 INFO:teuthology.orchestra.run.smithi038.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-26T06:47:38.848 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-26T06:47:38.848 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.849 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.849 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.849 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.849 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.849 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.849 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.849 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.849 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.849 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.850 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.850 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.850 INFO:teuthology.orchestra.run.smithi038.stdout: "ENABLE_BROWSABLE_API": { 2024-03-26T06:47:38.850 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-26T06:47:38.850 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.850 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.850 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.850 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.851 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.851 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.851 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.851 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.851 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.851 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.851 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.851 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.852 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-26T06:47:38.852 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-26T06:47:38.852 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.852 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.852 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.852 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.852 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.852 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.852 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.853 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.853 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.853 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.853 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.853 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.853 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-26T06:47:38.853 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-26T06:47:38.853 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.854 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.854 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.854 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.854 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.854 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.854 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.854 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.854 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.855 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.855 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.855 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.855 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-26T06:47:38.855 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-26T06:47:38.855 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.855 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.855 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.855 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.856 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.856 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.856 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.856 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.856 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.856 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.856 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.856 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.856 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-26T06:47:38.857 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-26T06:47:38.857 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.857 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.857 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.857 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.857 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.857 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.857 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.858 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.858 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.858 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.858 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.858 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.858 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-26T06:47:38.858 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-26T06:47:38.858 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.859 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.859 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.859 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.859 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.859 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.859 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.859 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.859 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.860 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.860 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.860 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.860 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-26T06:47:38.860 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-26T06:47:38.860 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.860 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.860 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.860 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.861 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.861 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.861 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.861 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.861 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.861 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.861 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.861 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.861 INFO:teuthology.orchestra.run.smithi038.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-26T06:47:38.862 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-26T06:47:38.862 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.862 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.862 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.862 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.862 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.862 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.862 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.862 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.863 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.863 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.863 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.863 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.863 INFO:teuthology.orchestra.run.smithi038.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-26T06:47:38.863 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-26T06:47:38.863 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.863 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.864 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.864 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.864 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.864 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.864 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.864 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.864 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.864 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.864 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.865 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.865 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_PASSWORD": { 2024-03-26T06:47:38.865 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-26T06:47:38.865 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.865 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.865 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.865 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-03-26T06:47:38.865 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.865 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.866 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.866 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.866 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.866 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.866 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.866 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.866 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-26T06:47:38.866 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-26T06:47:38.867 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.867 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.867 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.867 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.867 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.867 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.867 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.867 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.868 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.868 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.868 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.868 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.868 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_URL": { 2024-03-26T06:47:38.868 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_URL", 2024-03-26T06:47:38.868 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.868 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.869 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.869 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.869 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.869 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.869 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.869 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.869 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.869 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.870 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.870 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.870 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_API_USERNAME": { 2024-03-26T06:47:38.870 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-26T06:47:38.870 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.870 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.870 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.870 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-03-26T06:47:38.870 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.871 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.871 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.871 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.871 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.871 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.871 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.871 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.871 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-26T06:47:38.872 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-26T06:47:38.872 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.872 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.872 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.872 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.872 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.872 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.872 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.872 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.873 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.873 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.873 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.873 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.873 INFO:teuthology.orchestra.run.smithi038.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-26T06:47:38.873 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-26T06:47:38.873 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.873 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.873 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.874 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.874 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.874 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.874 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.874 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.874 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.874 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.874 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.875 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.875 INFO:teuthology.orchestra.run.smithi038.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-26T06:47:38.875 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-26T06:47:38.875 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.875 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.875 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.875 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.875 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.876 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.876 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.876 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.876 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.876 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.876 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.876 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.876 INFO:teuthology.orchestra.run.smithi038.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-26T06:47:38.876 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-26T06:47:38.877 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.877 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.877 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.877 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.877 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.877 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.877 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.877 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.878 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.878 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.878 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.878 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.878 INFO:teuthology.orchestra.run.smithi038.stdout: "MULTICLUSTER_CONFIG": { 2024-03-26T06:47:38.878 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-26T06:47:38.878 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.878 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.879 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.879 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "{}", 2024-03-26T06:47:38.879 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.879 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.879 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.879 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.879 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.879 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.880 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.880 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.880 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_HOST": { 2024-03-26T06:47:38.880 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-26T06:47:38.880 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.880 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.880 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.880 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.880 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.881 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.881 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.881 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.881 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.881 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.881 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.881 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.881 INFO:teuthology.orchestra.run.smithi038.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-26T06:47:38.881 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-26T06:47:38.882 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.882 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.882 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.882 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.882 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.882 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.882 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.882 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.883 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.883 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.883 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.883 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.883 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-26T06:47:38.883 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-26T06:47:38.883 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.883 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.883 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.884 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-26T06:47:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-26T06:47:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.885 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-26T06:47:38.886 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-26T06:47:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.887 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-26T06:47:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-26T06:47:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.888 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.889 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-26T06:47:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-26T06:47:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.890 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-26T06:47:38.891 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-26T06:47:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.892 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.893 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.893 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.893 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.893 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-26T06:47:38.893 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-26T06:47:38.893 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.893 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.893 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.893 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.894 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.894 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.894 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.894 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.894 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.894 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.894 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.894 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.895 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_ENABLED": { 2024-03-26T06:47:38.895 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-26T06:47:38.895 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.895 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.895 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.895 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.895 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.895 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.895 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.896 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.896 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.896 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.896 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.896 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.896 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-26T06:47:38.896 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-26T06:47:38.896 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.897 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.897 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.897 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-26T06:47:38.897 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.897 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.897 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.897 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.897 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.898 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.898 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.898 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.898 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-26T06:47:38.898 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-26T06:47:38.898 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.898 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.898 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.899 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.899 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.899 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.899 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.899 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.899 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.899 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.899 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.900 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.900 INFO:teuthology.orchestra.run.smithi038.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-26T06:47:38.900 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-26T06:47:38.900 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.900 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.900 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.900 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8", 2024-03-26T06:47:38.900 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.900 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.901 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.901 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.901 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.901 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.901 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.901 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.901 INFO:teuthology.orchestra.run.smithi038.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-26T06:47:38.901 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-26T06:47:38.901 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.902 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.902 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.902 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "45", 2024-03-26T06:47:38.902 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.902 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.902 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.902 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.902 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.903 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.903 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.903 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.903 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ACCESS_KEY": { 2024-03-26T06:47:38.903 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-26T06:47:38.903 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.903 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.903 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.904 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.904 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.904 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.904 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.904 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.904 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.904 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.904 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.904 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.905 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-26T06:47:38.905 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-26T06:47:38.905 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.905 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.905 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.905 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "admin", 2024-03-26T06:47:38.905 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.905 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.906 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.906 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.906 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.906 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.906 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.906 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.906 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SECRET_KEY": { 2024-03-26T06:47:38.907 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-26T06:47:38.907 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.907 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.907 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.907 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.907 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.907 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.907 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.908 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.908 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.908 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.908 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.908 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.908 INFO:teuthology.orchestra.run.smithi038.stdout: "RGW_API_SSL_VERIFY": { 2024-03-26T06:47:38.908 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-26T06:47:38.908 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.908 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.909 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.909 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.909 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.909 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.909 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.909 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.909 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.909 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.910 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.910 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.910 INFO:teuthology.orchestra.run.smithi038.stdout: "UNSAFE_TLS_v1_2": { 2024-03-26T06:47:38.910 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-26T06:47:38.910 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.910 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.910 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.910 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.910 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.911 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-26T06:47:38.912 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-26T06:47:38.912 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.912 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.912 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.912 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:38.912 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.912 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.912 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.912 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.913 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.913 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.913 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.913 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.913 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-26T06:47:38.913 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-26T06:47:38.913 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.913 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.913 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.914 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:38.914 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.914 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.914 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.914 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.914 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.915 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.915 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.915 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.915 INFO:teuthology.orchestra.run.smithi038.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-26T06:47:38.915 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-26T06:47:38.915 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.915 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.915 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.916 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:38.916 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.916 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.916 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.916 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.916 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.916 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.916 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.917 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.917 INFO:teuthology.orchestra.run.smithi038.stdout: "cross_origin_url": { 2024-03-26T06:47:38.917 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cross_origin_url", 2024-03-26T06:47:38.917 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.917 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.917 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.917 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.917 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.918 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.918 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.918 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.918 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.918 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.918 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.918 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.918 INFO:teuthology.orchestra.run.smithi038.stdout: "crt_file": { 2024-03-26T06:47:38.919 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "crt_file", 2024-03-26T06:47:38.919 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.919 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.919 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.919 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.919 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.919 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.919 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.919 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.920 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.920 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.920 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.920 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.920 INFO:teuthology.orchestra.run.smithi038.stdout: "debug": { 2024-03-26T06:47:38.920 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "debug", 2024-03-26T06:47:38.920 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.920 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.921 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.921 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.921 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.921 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.921 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.921 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Enable/disable debug options", 2024-03-26T06:47:38.921 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.921 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.921 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.922 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.922 INFO:teuthology.orchestra.run.smithi038.stdout: "jwt_token_ttl": { 2024-03-26T06:47:38.922 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "jwt_token_ttl", 2024-03-26T06:47:38.922 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.922 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.922 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.922 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "28800", 2024-03-26T06:47:38.922 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.923 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.923 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.923 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.923 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.923 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.923 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.923 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.923 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-03-26T06:47:38.923 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-03-26T06:47:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.924 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.925 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.926 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.927 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.928 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.929 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.930 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.931 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.932 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: "motd": { 2024-03-26T06:47:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "motd", 2024-03-26T06:47:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.933 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "The message of the day", 2024-03-26T06:47:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.934 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.935 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.935 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.935 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect_resolve_ip_addr": { 2024-03-26T06:47:38.935 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "redirect_resolve_ip_addr", 2024-03-26T06:47:38.935 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.935 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.935 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.935 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.936 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.936 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.936 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.936 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.936 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.936 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.936 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.936 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.937 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-03-26T06:47:38.937 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-03-26T06:47:38.937 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.937 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.937 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.937 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-03-26T06:47:38.937 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.937 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.938 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.938 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.938 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.938 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.938 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.938 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.938 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-03-26T06:47:38.938 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-03-26T06:47:38.939 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.939 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.939 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.939 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8080", 2024-03-26T06:47:38.939 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.939 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.939 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.939 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.939 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-03-26T06:47:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-03-26T06:47:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.940 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.941 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl_server_port": { 2024-03-26T06:47:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl_server_port", 2024-03-26T06:47:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8443", 2024-03-26T06:47:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.942 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-03-26T06:47:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-03-26T06:47:38.943 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.944 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.944 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.944 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "redirect", 2024-03-26T06:47:38.944 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.944 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.944 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.944 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.944 INFO:teuthology.orchestra.run.smithi038.stdout: "redirect" 2024-03-26T06:47:38.945 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.945 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.945 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.945 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.945 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.945 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.945 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-03-26T06:47:38.945 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-03-26T06:47:38.945 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.946 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.946 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.946 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-03-26T06:47:38.946 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-03-26T06:47:38.946 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-03-26T06:47:38.946 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.946 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.946 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.947 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.947 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.947 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.947 INFO:teuthology.orchestra.run.smithi038.stdout: "url_prefix": { 2024-03-26T06:47:38.947 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url_prefix", 2024-03-26T06:47:38.947 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.947 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.947 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.948 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.948 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.948 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.948 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.948 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.948 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.948 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.948 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.948 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.949 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.949 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.949 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.949 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-03-26T06:47:38.949 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.949 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.949 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.949 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_monitoring": { 2024-03-26T06:47:38.950 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_monitoring", 2024-03-26T06:47:38.950 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.950 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.950 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.950 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.950 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.950 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.950 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.951 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "monitor device health metrics", 2024-03-26T06:47:38.951 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.951 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.951 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.951 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.951 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.951 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.951 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.951 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.952 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.952 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.952 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.952 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.952 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.952 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.952 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.952 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.952 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.953 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.953 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.953 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.953 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.953 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.953 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.953 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.953 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.954 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.954 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.954 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.954 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.954 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.954 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.954 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.954 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.955 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.955 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.955 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.955 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.955 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.955 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.955 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.955 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.956 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.956 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.956 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.956 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.956 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.956 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.956 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.956 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.957 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.957 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.957 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.957 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.957 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.957 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.957 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.957 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.958 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.958 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.958 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.958 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.958 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.958 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.958 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.958 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.959 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.959 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.959 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.959 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.959 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.959 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.959 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.959 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.959 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.960 INFO:teuthology.orchestra.run.smithi038.stdout: "mark_out_threshold": { 2024-03-26T06:47:38.960 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mark_out_threshold", 2024-03-26T06:47:38.960 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.960 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.960 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.960 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "2419200", 2024-03-26T06:47:38.960 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.960 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.960 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.961 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-26T06:47:38.961 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.961 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.961 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.961 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.961 INFO:teuthology.orchestra.run.smithi038.stdout: "pool_name": { 2024-03-26T06:47:38.961 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pool_name", 2024-03-26T06:47:38.961 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.962 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.962 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.962 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "device_health_metrics", 2024-03-26T06:47:38.962 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.962 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.962 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.962 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-26T06:47:38.962 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.962 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.963 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.963 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.963 INFO:teuthology.orchestra.run.smithi038.stdout: "retention_period": { 2024-03-26T06:47:38.963 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "retention_period", 2024-03-26T06:47:38.963 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.963 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.963 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.963 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15552000", 2024-03-26T06:47:38.964 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.964 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.964 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.964 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long to retain device health metrics", 2024-03-26T06:47:38.964 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.964 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.964 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.964 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.964 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_frequency": { 2024-03-26T06:47:38.965 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_frequency", 2024-03-26T06:47:38.965 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.965 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.965 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.965 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-03-26T06:47:38.965 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.965 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.965 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.966 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-26T06:47:38.966 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.966 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.966 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.966 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.966 INFO:teuthology.orchestra.run.smithi038.stdout: "self_heal": { 2024-03-26T06:47:38.966 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "self_heal", 2024-03-26T06:47:38.966 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.967 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.967 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.967 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:38.967 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.967 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.967 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.967 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-26T06:47:38.967 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.968 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.968 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.968 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.968 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:38.968 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:38.968 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.968 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.968 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.969 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:38.969 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.969 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.969 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.969 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how frequently to wake up and check device health", 2024-03-26T06:47:38.969 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.969 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.969 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.969 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.970 INFO:teuthology.orchestra.run.smithi038.stdout: "warn_threshold": { 2024-03-26T06:47:38.970 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "warn_threshold", 2024-03-26T06:47:38.970 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.970 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.970 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.970 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7257600", 2024-03-26T06:47:38.970 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.970 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.971 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.971 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-26T06:47:38.971 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.971 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.971 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.971 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.971 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.971 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.972 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.972 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "diskprediction_local", 2024-03-26T06:47:38.972 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:38.972 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:38.972 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.972 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.972 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.972 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.972 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.973 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.973 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.973 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.973 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.973 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.973 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.973 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.973 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.974 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.974 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.974 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.974 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.974 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.974 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.974 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.974 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.974 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.975 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.975 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.975 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.975 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.975 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.975 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.975 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.975 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.976 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.976 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.976 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.976 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.976 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.976 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.976 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.976 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.976 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.977 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.977 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.977 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.977 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.977 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.977 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.977 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.977 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.978 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.978 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.978 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.978 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.978 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.978 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.978 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.978 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.979 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.979 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.979 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:38.979 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:38.979 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.979 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.979 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.979 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.980 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.980 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.980 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.980 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.980 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.980 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.980 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.980 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.980 INFO:teuthology.orchestra.run.smithi038.stdout: "predict_interval": { 2024-03-26T06:47:38.981 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predict_interval", 2024-03-26T06:47:38.981 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.981 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.981 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.981 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "86400", 2024-03-26T06:47:38.981 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.981 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.981 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.981 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.982 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.982 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.982 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.982 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.982 INFO:teuthology.orchestra.run.smithi038.stdout: "predictor_model": { 2024-03-26T06:47:38.982 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "predictor_model", 2024-03-26T06:47:38.982 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.982 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.983 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.983 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "prophetstor", 2024-03-26T06:47:38.983 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.983 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.983 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.983 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.983 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.983 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.984 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.984 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:38.984 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:38.984 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.984 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.984 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.984 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "600", 2024-03-26T06:47:38.984 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.985 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.985 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.985 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.985 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.985 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.985 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.985 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:38.986 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.986 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:38.986 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "influx", 2024-03-26T06:47:38.986 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": false, 2024-03-26T06:47:38.986 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "influxdb python module not found", 2024-03-26T06:47:38.986 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:38.986 INFO:teuthology.orchestra.run.smithi038.stdout: "batch_size": { 2024-03-26T06:47:38.986 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "batch_size", 2024-03-26T06:47:38.987 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:38.987 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.987 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.987 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5000", 2024-03-26T06:47:38.987 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.987 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.987 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.987 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-26T06:47:38.988 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.988 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.988 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.988 INFO:teuthology.orchestra.run.smithi038.stdout: "database": { 2024-03-26T06:47:38.988 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "database", 2024-03-26T06:47:38.988 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.988 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.989 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.989 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "ceph", 2024-03-26T06:47:38.989 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.989 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.989 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.989 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-26T06:47:38.989 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.989 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.989 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.990 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.990 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": { 2024-03-26T06:47:38.990 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "hostname", 2024-03-26T06:47:38.990 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.990 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.990 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.990 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.990 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.991 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.991 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.991 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server hostname", 2024-03-26T06:47:38.991 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.991 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.991 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.991 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:38.992 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:38.992 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:38.992 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.992 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:38.992 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "30", 2024-03-26T06:47:38.992 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "5", 2024-03-26T06:47:38.992 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.992 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.992 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-26T06:47:38.993 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.993 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.993 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.993 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:38.993 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:38.993 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.993 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.993 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.994 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:38.994 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.994 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.994 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.994 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.994 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.994 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.994 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.995 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.995 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.995 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.995 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.995 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.995 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.995 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.995 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.995 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:38.996 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:38.996 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:38.996 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.996 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.996 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:38.996 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.996 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.996 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:38.997 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.997 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.997 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.997 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:38.997 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:38.997 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:38.997 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:38.997 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:38.998 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:38.998 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:38.998 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:38.998 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:38.998 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:38.998 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:38.998 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:38.998 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:38.999 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.000 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.000 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.000 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.000 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.000 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.000 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.000 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.000 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.000 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.001 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.001 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.001 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.001 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.001 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.001 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.001 INFO:teuthology.orchestra.run.smithi038.stdout: "password": { 2024-03-26T06:47:39.001 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "password", 2024-03-26T06:47:39.002 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.002 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.002 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.002 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.002 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.002 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.002 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.002 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "password of InfluxDB server user", 2024-03-26T06:47:39.002 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.003 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.003 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.003 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.003 INFO:teuthology.orchestra.run.smithi038.stdout: "port": { 2024-03-26T06:47:39.003 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "port", 2024-03-26T06:47:39.003 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.003 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.003 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.004 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "8086", 2024-03-26T06:47:39.004 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.004 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.004 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.004 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "InfluxDB server port", 2024-03-26T06:47:39.004 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.004 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.004 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.004 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.005 INFO:teuthology.orchestra.run.smithi038.stdout: "ssl": { 2024-03-26T06:47:39.005 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ssl", 2024-03-26T06:47:39.005 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.005 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.005 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.005 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "false", 2024-03-26T06:47:39.005 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.005 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.006 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.006 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-26T06:47:39.006 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.006 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.006 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.006 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.006 INFO:teuthology.orchestra.run.smithi038.stdout: "threads": { 2024-03-26T06:47:39.006 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threads", 2024-03-26T06:47:39.007 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.007 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.007 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.007 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:39.007 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:39.007 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "32", 2024-03-26T06:47:39.007 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.007 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-26T06:47:39.008 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.008 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.008 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.008 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.008 INFO:teuthology.orchestra.run.smithi038.stdout: "username": { 2024-03-26T06:47:39.008 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "username", 2024-03-26T06:47:39.008 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.008 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.009 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.009 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.009 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.009 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.009 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.009 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "username of InfluxDB server user", 2024-03-26T06:47:39.009 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.009 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.009 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.010 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.010 INFO:teuthology.orchestra.run.smithi038.stdout: "verify_ssl": { 2024-03-26T06:47:39.010 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "verify_ssl", 2024-03-26T06:47:39.010 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.010 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.010 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.010 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "true", 2024-03-26T06:47:39.010 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.011 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.011 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.011 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-26T06:47:39.011 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.011 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.011 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.011 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.011 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.011 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.012 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.012 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "insights", 2024-03-26T06:47:39.012 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.012 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.012 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.012 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.012 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.012 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.013 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.013 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.013 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.013 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.013 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.013 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.013 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.013 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.013 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.014 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.014 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.014 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.014 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.014 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.014 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.014 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.014 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.015 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.015 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.015 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.015 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.015 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.015 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.015 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.015 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.016 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.017 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.018 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.018 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.018 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.018 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.018 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.018 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.018 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.018 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.019 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.019 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.019 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.019 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.019 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.019 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.019 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.019 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.020 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.020 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.020 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.020 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.020 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.020 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.020 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.020 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.020 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.021 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.021 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.021 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.021 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "iostat", 2024-03-26T06:47:39.021 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.021 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.021 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.021 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.021 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.022 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.022 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.022 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.022 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.022 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.022 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.022 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.022 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.023 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.023 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.023 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.023 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.023 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.023 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.023 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.023 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.024 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.024 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.024 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.024 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.024 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.024 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.026 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.026 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.026 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.026 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.026 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.026 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.026 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.026 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.027 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.027 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.027 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.027 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.027 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.027 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.027 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.027 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.028 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.028 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.028 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.028 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.028 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.028 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.028 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.028 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.029 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.029 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.029 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.029 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.029 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.029 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.029 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.029 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.029 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.030 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.030 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.030 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.030 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.030 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.030 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.030 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.030 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.031 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.031 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.031 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.031 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.031 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.031 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.031 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.031 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "k8sevents", 2024-03-26T06:47:39.032 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.032 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.032 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.032 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_event_retention_days": { 2024-03-26T06:47:39.032 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "ceph_event_retention_days", 2024-03-26T06:47:39.032 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.032 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.032 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.033 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "7", 2024-03-26T06:47:39.033 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.033 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.033 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.033 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-26T06:47:39.033 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.033 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.033 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.033 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.034 INFO:teuthology.orchestra.run.smithi038.stdout: "config_check_secs": { 2024-03-26T06:47:39.034 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "config_check_secs", 2024-03-26T06:47:39.034 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.034 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.034 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.034 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:39.034 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "10", 2024-03-26T06:47:39.034 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.035 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.035 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-26T06:47:39.035 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.035 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.035 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.035 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.035 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.035 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.035 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.036 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.036 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.036 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.036 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.036 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.036 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.036 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.036 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.036 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.037 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.037 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.037 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.037 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.037 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.037 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.037 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.037 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.038 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.038 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.038 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.038 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.038 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.038 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.038 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.038 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.039 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.039 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.039 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.039 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.039 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.039 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.039 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.039 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.039 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.040 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.040 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.040 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.040 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.040 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.040 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.040 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.040 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.041 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.041 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.041 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.041 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.041 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.041 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.041 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.041 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.042 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.042 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.042 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.042 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.042 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.042 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.042 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.042 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.043 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.043 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.043 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.043 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.043 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.043 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.043 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.043 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.043 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.044 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.044 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.044 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.044 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "localpool", 2024-03-26T06:47:39.044 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.044 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.044 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.044 INFO:teuthology.orchestra.run.smithi038.stdout: "failure_domain": { 2024-03-26T06:47:39.044 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "failure_domain", 2024-03-26T06:47:39.045 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.045 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.045 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.045 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "host", 2024-03-26T06:47:39.045 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.045 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.045 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.045 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "failure domain for any created local pool", 2024-03-26T06:47:39.046 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-26T06:47:39.046 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.046 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.046 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.046 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.046 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.046 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.046 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.047 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.047 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.047 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.047 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.047 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.047 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.047 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.047 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.048 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.048 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.048 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.048 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.048 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.048 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.048 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.048 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.048 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.049 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.049 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.049 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.049 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.049 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.049 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.049 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.049 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.049 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.050 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.050 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.050 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.050 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.050 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.050 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.050 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.050 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.051 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.051 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.051 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.051 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.051 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.051 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.051 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.051 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.052 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.052 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.052 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.052 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.052 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.052 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.052 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.052 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.053 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.053 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.053 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.053 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.053 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.053 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.053 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.053 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.053 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.054 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.054 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.054 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.054 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.054 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.054 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.054 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.054 INFO:teuthology.orchestra.run.smithi038.stdout: "min_size": { 2024-03-26T06:47:39.054 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "min_size", 2024-03-26T06:47:39.055 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.055 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.055 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.055 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.055 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.055 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.055 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.055 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default min_size for any created local pool", 2024-03-26T06:47:39.056 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-26T06:47:39.056 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.056 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.056 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.056 INFO:teuthology.orchestra.run.smithi038.stdout: "num_rep": { 2024-03-26T06:47:39.056 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "num_rep", 2024-03-26T06:47:39.056 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.056 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.056 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.057 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3", 2024-03-26T06:47:39.057 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.057 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.057 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.057 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default replica count for any created local pool", 2024-03-26T06:47:39.057 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.057 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.058 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.058 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.058 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_num": { 2024-03-26T06:47:39.058 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_num", 2024-03-26T06:47:39.058 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.058 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.058 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.058 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "128", 2024-03-26T06:47:39.059 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.059 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.059 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.059 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "default pg_num for any created local pool", 2024-03-26T06:47:39.059 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.059 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.059 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.059 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.059 INFO:teuthology.orchestra.run.smithi038.stdout: "prefix": { 2024-03-26T06:47:39.060 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prefix", 2024-03-26T06:47:39.060 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.060 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.060 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.060 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.060 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.060 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.060 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.061 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "name prefix for any created local pool", 2024-03-26T06:47:39.061 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.061 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.061 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.061 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.061 INFO:teuthology.orchestra.run.smithi038.stdout: "subtree": { 2024-03-26T06:47:39.061 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "subtree", 2024-03-26T06:47:39.061 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.062 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.062 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.062 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "rack", 2024-03-26T06:47:39.062 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.062 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.062 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.062 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-26T06:47:39.062 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-26T06:47:39.062 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.063 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.063 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.063 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.063 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.063 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mds_autoscaler", 2024-03-26T06:47:39.063 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.063 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.064 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.064 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.064 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.064 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.064 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.064 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.064 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.064 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.064 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.065 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.065 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.065 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.065 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.065 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.065 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.065 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.065 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.066 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.066 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.066 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.066 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.066 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.066 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.066 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.066 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.066 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.067 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.067 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.067 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.067 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.067 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.067 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.067 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.068 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.068 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.068 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.068 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.068 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.068 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.068 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.068 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.068 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.069 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.069 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.069 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.069 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.069 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.069 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.069 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.069 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.070 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.070 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.070 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.070 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.070 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.070 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.070 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.070 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.071 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.071 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.071 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.071 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.071 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.071 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.071 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.071 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.071 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.072 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.072 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.072 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.072 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.072 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.072 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.072 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.072 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirroring", 2024-03-26T06:47:39.073 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.073 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.073 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.073 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.073 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.073 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.073 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.073 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.073 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.074 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.074 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.074 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.074 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.074 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.074 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.074 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.074 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.075 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.075 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.075 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.075 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.075 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.075 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.075 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.075 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.075 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.076 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.076 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.076 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.076 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.076 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.076 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.076 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.076 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.077 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.077 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.077 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.077 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.077 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.077 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.077 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.077 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.078 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.078 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.078 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.078 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.078 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.078 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.078 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.078 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.079 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.079 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.079 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.079 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.079 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.079 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.079 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.079 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.080 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.080 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.080 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.080 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.080 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.080 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.080 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.080 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.080 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.081 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.081 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.081 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.081 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.081 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.081 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.081 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.082 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.082 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "nfs", 2024-03-26T06:47:39.082 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.082 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.082 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.082 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.082 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.082 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.083 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.083 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.083 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.083 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.083 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.083 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.083 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.083 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.084 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.084 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.084 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.084 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.084 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.084 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.084 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.084 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.084 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.085 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.085 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.085 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.085 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.085 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.085 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.085 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.085 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.085 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.086 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.086 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.086 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.086 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.086 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.086 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.086 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.086 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.087 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.087 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.087 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.087 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.087 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.087 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.087 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.087 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.088 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.088 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.088 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.088 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.088 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.088 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.088 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.088 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.089 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.089 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.089 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.089 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.089 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.089 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.089 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.089 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.089 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.090 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.090 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.090 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.090 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.090 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.090 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.090 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.090 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.091 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.091 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.091 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.091 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-03-26T06:47:39.091 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.091 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.091 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.091 INFO:teuthology.orchestra.run.smithi038.stdout: "fail_fs": { 2024-03-26T06:47:39.091 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "fail_fs", 2024-03-26T06:47:39.092 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.092 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.092 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.092 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.092 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.092 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.092 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.092 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-26T06:47:39.092 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.093 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.093 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.093 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.093 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.093 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.093 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.093 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.093 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.094 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.095 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.096 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.097 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.098 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.099 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.100 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator": { 2024-03-26T06:47:39.101 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "orchestrator", 2024-03-26T06:47:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.102 INFO:teuthology.orchestra.run.smithi038.stdout: "cephadm", 2024-03-26T06:47:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "rook", 2024-03-26T06:47:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "test_orchestrator" 2024-03-26T06:47:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Orchestrator backend", 2024-03-26T06:47:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.103 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_perf_query", 2024-03-26T06:47:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.104 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.105 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.106 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.107 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.108 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.109 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.110 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.111 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.112 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "osd_support", 2024-03-26T06:47:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.113 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.114 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.115 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.116 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.116 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.116 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.116 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.116 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.116 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.116 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.116 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.117 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.117 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.117 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.117 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.117 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.117 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.117 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.117 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.118 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.118 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.118 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.118 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.118 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.118 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.118 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.118 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.119 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.119 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.119 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.119 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.119 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.119 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.119 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.119 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.119 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.119 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.120 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.120 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.120 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.120 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.120 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.120 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.120 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.120 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.121 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.121 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.121 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.121 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.121 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.121 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.121 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.121 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.122 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.122 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.122 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.122 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.122 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.122 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.122 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "pg_autoscaler", 2024-03-26T06:47:39.122 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.123 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.123 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.123 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.123 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.123 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.123 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.123 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.123 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.123 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.124 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.124 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.124 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.124 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.124 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.124 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.124 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.124 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.124 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.125 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.125 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.125 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.125 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.125 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.125 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.125 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.125 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.125 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.126 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.126 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.126 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.126 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.126 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.126 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.126 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.126 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.127 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.127 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.127 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.127 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.127 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.127 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.127 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.127 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.128 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.128 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.128 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.128 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.128 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.128 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.128 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.128 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.129 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.129 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.129 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.129 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.129 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.129 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.129 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.129 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.129 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.130 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.130 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.130 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.130 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.130 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.130 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.130 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.130 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.131 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.131 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.131 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.131 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.131 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:39.131 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:39.131 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:39.131 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.131 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.132 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:39.132 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.132 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.132 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.132 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.132 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.132 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.132 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.133 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.133 INFO:teuthology.orchestra.run.smithi038.stdout: "threshold": { 2024-03-26T06:47:39.133 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "threshold", 2024-03-26T06:47:39.133 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:39.133 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.133 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.133 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "3.0", 2024-03-26T06:47:39.133 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1.0", 2024-03-26T06:47:39.133 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.134 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.134 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "scaling threshold", 2024-03-26T06:47:39.134 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-26T06:47:39.134 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.134 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.134 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.134 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.135 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.135 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "progress", 2024-03-26T06:47:39.135 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.135 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.135 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.135 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_pg_recovery_event": { 2024-03-26T06:47:39.135 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_pg_recovery_event", 2024-03-26T06:47:39.135 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.136 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.136 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.136 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.136 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.136 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.136 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.136 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-26T06:47:39.136 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.137 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.137 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.137 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.137 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-03-26T06:47:39.137 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-03-26T06:47:39.137 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.137 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.137 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.138 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:39.138 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.138 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.138 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.138 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.138 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.138 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.138 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.139 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.139 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.139 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.139 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.139 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.139 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.139 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.139 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.139 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.140 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.140 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.140 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.140 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.140 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.140 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.140 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.140 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.140 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.141 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.141 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.141 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.141 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.141 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.141 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.141 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.141 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.142 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.142 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.142 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.142 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.142 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.142 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.142 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.142 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.143 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.143 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.143 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.143 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.143 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.143 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.143 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.143 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.144 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.144 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.144 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.144 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.144 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.144 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.144 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.144 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.144 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.145 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.145 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.145 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.145 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.145 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.145 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.145 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.145 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.146 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.146 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.146 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.146 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.146 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.146 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.146 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.146 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.146 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.147 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.147 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.147 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.147 INFO:teuthology.orchestra.run.smithi038.stdout: "max_completed_events": { 2024-03-26T06:47:39.147 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_completed_events", 2024-03-26T06:47:39.147 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.147 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.147 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.148 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "50", 2024-03-26T06:47:39.148 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.148 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.148 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.148 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "number of past completed events to remember", 2024-03-26T06:47:39.148 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.148 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.148 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.149 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.149 INFO:teuthology.orchestra.run.smithi038.stdout: "sleep_interval": { 2024-03-26T06:47:39.149 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "sleep_interval", 2024-03-26T06:47:39.149 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:39.149 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.149 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.149 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "5", 2024-03-26T06:47:39.149 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.149 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.150 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.150 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "how long the module is going to sleep", 2024-03-26T06:47:39.150 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.150 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.150 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.150 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.150 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.151 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.151 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "prometheus", 2024-03-26T06:47:39.151 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.151 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.151 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.151 INFO:teuthology.orchestra.run.smithi038.stdout: "cache": { 2024-03-26T06:47:39.151 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "cache", 2024-03-26T06:47:39.151 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.151 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.152 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.152 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:39.152 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.152 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.152 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.152 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.152 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.152 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.153 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.153 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.153 INFO:teuthology.orchestra.run.smithi038.stdout: "exclude_perf_counters": { 2024-03-26T06:47:39.153 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "exclude_perf_counters", 2024-03-26T06:47:39.153 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.153 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.153 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.153 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:39.154 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.154 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.154 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.154 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-26T06:47:39.154 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-26T06:47:39.154 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.154 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.154 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.154 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.155 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.155 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.155 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.155 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.155 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.155 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.155 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.155 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.156 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.156 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.156 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.156 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.156 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.156 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.156 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.156 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.156 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.157 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.157 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.157 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.157 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.157 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.157 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.157 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.158 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.158 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.158 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.158 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.158 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.158 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.158 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.158 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.159 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.159 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.159 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.159 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.159 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.159 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.159 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.159 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.159 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.160 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.160 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.160 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.160 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.160 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.160 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.160 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.160 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.161 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.161 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.161 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.161 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.161 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.161 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.161 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.161 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.162 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.163 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.163 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.163 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.163 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools": { 2024-03-26T06:47:39.163 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools", 2024-03-26T06:47:39.163 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.163 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.163 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.164 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.164 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.164 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.164 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.164 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.164 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.164 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.164 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.165 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.165 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-26T06:47:39.165 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-26T06:47:39.165 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.165 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.165 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.165 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "300", 2024-03-26T06:47:39.165 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.165 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.166 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.166 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.166 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.166 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.166 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.166 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.166 INFO:teuthology.orchestra.run.smithi038.stdout: "scrape_interval": { 2024-03-26T06:47:39.166 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "scrape_interval", 2024-03-26T06:47:39.167 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:39.167 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.167 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.167 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15.0", 2024-03-26T06:47:39.167 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.167 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.167 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.168 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.168 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.168 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.168 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.168 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.168 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-03-26T06:47:39.168 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-03-26T06:47:39.168 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.168 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.169 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.169 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "::", 2024-03-26T06:47:39.169 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.169 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.169 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.169 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-26T06:47:39.169 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.169 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.170 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.170 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.170 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-03-26T06:47:39.170 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-03-26T06:47:39.170 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.170 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.170 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.170 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "9283", 2024-03-26T06:47:39.171 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.171 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.171 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.171 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-26T06:47:39.171 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.171 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.171 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.171 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.171 INFO:teuthology.orchestra.run.smithi038.stdout: "stale_cache_strategy": { 2024-03-26T06:47:39.172 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stale_cache_strategy", 2024-03-26T06:47:39.172 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.172 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.172 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.172 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "log", 2024-03-26T06:47:39.172 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.172 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.172 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.173 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.173 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.173 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.173 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.173 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.173 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_behaviour": { 2024-03-26T06:47:39.173 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_behaviour", 2024-03-26T06:47:39.173 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.173 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.174 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.174 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "default", 2024-03-26T06:47:39.174 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.174 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.174 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.174 INFO:teuthology.orchestra.run.smithi038.stdout: "default", 2024-03-26T06:47:39.174 INFO:teuthology.orchestra.run.smithi038.stdout: "error" 2024-03-26T06:47:39.174 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.175 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.175 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.175 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.175 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.175 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.175 INFO:teuthology.orchestra.run.smithi038.stdout: "standby_error_status_code": { 2024-03-26T06:47:39.175 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "standby_error_status_code", 2024-03-26T06:47:39.175 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.175 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.176 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.176 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "500", 2024-03-26T06:47:39.176 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "400", 2024-03-26T06:47:39.176 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "599", 2024-03-26T06:47:39.176 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.176 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.176 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.176 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.176 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.177 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.177 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.177 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.177 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.177 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-03-26T06:47:39.177 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.177 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.178 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.178 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.178 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.178 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.178 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.178 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.178 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.178 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.179 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.179 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.179 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.179 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.179 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.179 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.179 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.179 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.179 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.180 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.180 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.180 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.180 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.180 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.180 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.180 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.180 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.181 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.181 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.181 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.181 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.181 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.181 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.181 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.181 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.182 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.182 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.182 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.182 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.182 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.182 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.182 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.182 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.183 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.183 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.183 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.183 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.183 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.183 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.183 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.183 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.183 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.184 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.184 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.184 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.184 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.184 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.184 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.184 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.184 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.185 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.185 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.185 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.185 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.185 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.185 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.185 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.185 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.185 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_snap_create": { 2024-03-26T06:47:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_snap_create", 2024-03-26T06:47:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.186 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10", 2024-03-26T06:47:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.187 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "mirror_snapshot_schedule": { 2024-03-26T06:47:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "mirror_snapshot_schedule", 2024-03-26T06:47:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.188 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "trash_purge_schedule": { 2024-03-26T06:47:39.189 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "trash_purge_schedule", 2024-03-26T06:47:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.190 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "restful", 2024-03-26T06:47:39.191 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "enable_auth": { 2024-03-26T06:47:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enable_auth", 2024-03-26T06:47:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.192 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.193 INFO:teuthology.orchestra.run.smithi038.stdout: "key_file": { 2024-03-26T06:47:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "key_file", 2024-03-26T06:47:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.194 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.195 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.195 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.195 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.195 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.195 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.195 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.195 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.195 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.195 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.196 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.196 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.196 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.196 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.196 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.196 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.196 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.196 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.197 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.197 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.197 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.197 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.197 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.197 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.197 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.197 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.198 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.198 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.198 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.198 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.198 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.198 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.198 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.198 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.198 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.199 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.199 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.199 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.199 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.199 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.199 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.199 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.199 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.200 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.200 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.200 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.200 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.200 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.200 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.200 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.200 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.200 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.201 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.201 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.201 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.201 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.201 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.201 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.201 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.201 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.202 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.202 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.202 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.202 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.202 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.202 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.202 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.202 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.202 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.203 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.203 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.203 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.203 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.203 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.203 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.203 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.203 INFO:teuthology.orchestra.run.smithi038.stdout: "server_addr": { 2024-03-26T06:47:39.204 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_addr", 2024-03-26T06:47:39.204 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.204 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.204 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.204 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.204 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.204 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.204 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.204 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.205 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.205 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.205 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.205 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.205 INFO:teuthology.orchestra.run.smithi038.stdout: "server_port": { 2024-03-26T06:47:39.205 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "server_port", 2024-03-26T06:47:39.205 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.205 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.205 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.206 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.206 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.206 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.206 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.206 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.206 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.206 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.206 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.207 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.207 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.207 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.207 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.207 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rgw", 2024-03-26T06:47:39.207 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.207 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.207 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.208 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.208 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.208 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.208 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.208 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.208 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.208 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.208 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.208 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.209 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.209 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.209 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.209 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.209 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.209 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.209 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.209 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.210 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.210 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.210 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.210 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.210 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.210 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.210 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.210 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.211 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.211 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.211 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.211 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.211 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.211 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.211 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.211 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.211 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.212 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.212 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.212 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.212 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.212 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.212 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.212 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.212 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.213 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.213 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.213 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.213 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.213 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.213 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.213 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.213 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.213 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.214 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.214 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.214 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.214 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.214 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.214 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.214 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.214 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.215 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.215 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.215 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.215 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.215 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.215 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.215 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.215 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.216 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.216 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.216 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.216 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.216 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.216 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rook", 2024-03-26T06:47:39.216 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.216 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.217 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.217 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.217 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.217 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.217 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.217 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.217 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.217 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.218 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.218 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.218 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.218 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.218 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.218 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.218 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.219 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.219 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.219 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.219 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.219 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.219 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.219 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.219 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.219 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.220 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.220 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.220 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.220 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.220 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.220 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.220 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.220 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.221 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.221 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.221 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.221 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.221 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.221 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.221 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.221 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.221 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.222 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.222 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.222 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.222 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.222 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.222 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.222 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.222 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.223 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.223 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.223 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.223 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.223 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.223 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.223 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.223 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.223 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.224 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.224 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.224 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.224 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.224 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.224 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.224 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.224 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.225 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.225 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.225 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.225 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.225 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.225 INFO:teuthology.orchestra.run.smithi038.stdout: "storage_class": { 2024-03-26T06:47:39.225 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "storage_class", 2024-03-26T06:47:39.225 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.225 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.226 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.226 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "local", 2024-03-26T06:47:39.226 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.226 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.226 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.226 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-26T06:47:39.226 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.226 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.227 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.227 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.227 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.227 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.227 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.227 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "selftest", 2024-03-26T06:47:39.227 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.227 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.228 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.228 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.228 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.228 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.228 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.228 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.228 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.228 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.229 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.229 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.229 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.229 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.229 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.229 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.229 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.229 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.230 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.230 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.230 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.230 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.230 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.230 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.230 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.230 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.230 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.231 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.231 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.231 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.231 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.231 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.231 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.231 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.231 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.231 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.232 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.232 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.232 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.232 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.232 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.232 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.232 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.232 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.233 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.233 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.233 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.233 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.233 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.233 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.233 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.233 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.233 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.234 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.234 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.234 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.234 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.234 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.234 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.234 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.234 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.235 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.235 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.235 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.235 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.235 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.235 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.235 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.235 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.236 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.236 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.236 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.236 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.236 INFO:teuthology.orchestra.run.smithi038.stdout: "roption1": { 2024-03-26T06:47:39.236 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption1", 2024-03-26T06:47:39.236 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.236 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.236 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.237 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.237 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.237 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.237 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.237 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.237 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.237 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.237 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.238 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.238 INFO:teuthology.orchestra.run.smithi038.stdout: "roption2": { 2024-03-26T06:47:39.238 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "roption2", 2024-03-26T06:47:39.238 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.238 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.238 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.238 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "xyz", 2024-03-26T06:47:39.239 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.239 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.239 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.239 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.239 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.239 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.239 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.239 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.239 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption1": { 2024-03-26T06:47:39.240 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption1", 2024-03-26T06:47:39.240 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.240 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.240 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.240 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.240 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.240 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.240 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.240 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.241 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.241 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.241 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.241 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.241 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption2": { 2024-03-26T06:47:39.241 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption2", 2024-03-26T06:47:39.241 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.241 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.241 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.242 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.242 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.242 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.242 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.242 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.242 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.242 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.242 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.243 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.243 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption3": { 2024-03-26T06:47:39.243 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption3", 2024-03-26T06:47:39.243 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "float", 2024-03-26T06:47:39.243 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.243 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.243 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.243 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.243 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.244 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.244 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.244 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.244 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.244 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.244 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.244 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption4": { 2024-03-26T06:47:39.244 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption4", 2024-03-26T06:47:39.245 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.245 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.245 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.245 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.245 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.245 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.245 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.245 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.245 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.246 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.246 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.246 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.246 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption5": { 2024-03-26T06:47:39.246 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption5", 2024-03-26T06:47:39.246 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.246 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.246 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.247 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.247 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.247 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.247 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.247 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.247 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.247 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.247 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.248 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.248 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption6": { 2024-03-26T06:47:39.248 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption6", 2024-03-26T06:47:39.248 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.248 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.248 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.248 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:39.248 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.249 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.249 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.249 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.249 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.249 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.249 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.249 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.249 INFO:teuthology.orchestra.run.smithi038.stdout: "rwoption7": { 2024-03-26T06:47:39.249 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rwoption7", 2024-03-26T06:47:39.250 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.250 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.250 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.250 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.250 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "1", 2024-03-26T06:47:39.250 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "42", 2024-03-26T06:47:39.250 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.250 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.250 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.251 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.251 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.251 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.251 INFO:teuthology.orchestra.run.smithi038.stdout: "testkey": { 2024-03-26T06:47:39.251 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testkey", 2024-03-26T06:47:39.251 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.251 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.251 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.252 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.252 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.252 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.252 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.252 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.252 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.252 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.252 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.252 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.253 INFO:teuthology.orchestra.run.smithi038.stdout: "testlkey": { 2024-03-26T06:47:39.253 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testlkey", 2024-03-26T06:47:39.253 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.253 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.253 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.253 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.253 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.253 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.253 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.254 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.254 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.254 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.254 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.254 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.254 INFO:teuthology.orchestra.run.smithi038.stdout: "testnewline": { 2024-03-26T06:47:39.254 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "testnewline", 2024-03-26T06:47:39.254 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.255 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.255 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.255 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.255 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.255 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.255 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.255 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.255 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.255 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.256 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.256 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.256 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.256 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.256 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-03-26T06:47:39.256 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.256 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.257 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.257 INFO:teuthology.orchestra.run.smithi038.stdout: "allow_m_granularity": { 2024-03-26T06:47:39.257 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "allow_m_granularity", 2024-03-26T06:47:39.257 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.257 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.257 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.257 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.257 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.258 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.258 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.258 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "allow minute scheduled snapshots", 2024-03-26T06:47:39.258 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.258 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.258 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.258 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.258 INFO:teuthology.orchestra.run.smithi038.stdout: "dump_on_update": { 2024-03-26T06:47:39.258 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "dump_on_update", 2024-03-26T06:47:39.259 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.259 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.259 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.259 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.259 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.259 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.259 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.259 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "dump database to debug log on update", 2024-03-26T06:47:39.260 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.260 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.260 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.260 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.260 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.260 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.260 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.260 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.260 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.261 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.261 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.261 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.261 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.261 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.261 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.261 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.261 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.262 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.262 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.262 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.262 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.262 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.262 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.262 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.262 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.262 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.263 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.264 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.265 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.266 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.267 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.268 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "stats", 2024-03-26T06:47:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.269 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.270 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.271 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.272 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.272 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.272 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.272 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.272 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.272 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.272 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.272 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.273 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.273 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.273 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.273 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.273 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.273 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.273 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.273 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.273 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.274 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.274 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.274 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.274 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.274 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.274 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.274 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.274 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.275 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.275 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.275 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.275 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.275 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.275 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.275 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.275 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.275 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.276 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.276 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.276 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.276 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.276 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.276 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.276 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.276 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.277 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.277 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.277 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.277 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.277 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.277 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.277 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.277 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.278 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.278 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.278 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.278 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "status", 2024-03-26T06:47:39.278 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.278 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.278 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.278 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.279 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.279 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.279 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.279 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.279 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.279 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.279 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.279 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.279 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.280 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.280 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.280 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.280 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.280 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.280 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.280 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.280 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.280 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.281 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.281 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.281 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.281 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.281 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.281 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.281 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.281 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.282 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.282 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.282 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.282 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.282 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.282 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.282 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.282 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.283 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.283 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.283 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.283 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.283 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.283 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.283 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.283 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.284 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.284 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.284 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.284 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.284 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.284 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.284 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.284 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.284 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.285 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.285 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.285 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.285 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.285 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.285 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.285 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.285 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.286 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.286 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.286 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.286 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.286 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.286 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.286 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.286 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.286 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.287 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.287 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.287 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.287 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telegraf", 2024-03-26T06:47:39.287 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.287 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.287 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.288 INFO:teuthology.orchestra.run.smithi038.stdout: "address": { 2024-03-26T06:47:39.288 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "address", 2024-03-26T06:47:39.288 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.288 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.288 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.288 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-26T06:47:39.288 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.288 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.289 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.289 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.289 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.289 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.289 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.289 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.289 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:39.289 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:39.289 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:39.290 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.290 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.290 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "15", 2024-03-26T06:47:39.290 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.290 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.290 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.290 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.290 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.291 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.291 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.291 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.291 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.291 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.291 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.291 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.291 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.291 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.292 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.292 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.292 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.292 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.292 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.292 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.292 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.292 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.292 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.293 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.293 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.293 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.293 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.293 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.293 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.293 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.293 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.294 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.294 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.294 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.294 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.294 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.294 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.294 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.294 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.295 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.295 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.295 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.295 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.295 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.295 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.295 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.295 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.295 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.296 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.296 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.296 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.296 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.296 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.296 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.296 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.296 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.297 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.297 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.297 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.297 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.297 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.297 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.297 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.298 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.298 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.298 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.298 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.298 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.298 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.298 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.298 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.299 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.299 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.299 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.299 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.299 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.299 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.299 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "telemetry", 2024-03-26T06:47:39.300 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.300 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.300 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.300 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_basic": { 2024-03-26T06:47:39.300 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_basic", 2024-03-26T06:47:39.300 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.300 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.300 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.301 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:39.301 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.301 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.301 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.301 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-26T06:47:39.301 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.301 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.301 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.302 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_crash": { 2024-03-26T06:47:39.302 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_crash", 2024-03-26T06:47:39.302 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.302 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.302 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.302 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:39.302 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.302 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.303 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.303 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-26T06:47:39.303 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.303 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.303 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.303 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.303 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_device": { 2024-03-26T06:47:39.303 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_device", 2024-03-26T06:47:39.303 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.304 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.304 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.304 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:39.304 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.304 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.304 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.304 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-26T06:47:39.304 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.304 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.305 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.305 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.305 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_ident": { 2024-03-26T06:47:39.305 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_ident", 2024-03-26T06:47:39.305 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.305 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.305 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.305 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.305 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.306 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.306 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.306 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-26T06:47:39.306 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.306 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.306 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.306 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.307 INFO:teuthology.orchestra.run.smithi038.stdout: "channel_perf": { 2024-03-26T06:47:39.307 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "channel_perf", 2024-03-26T06:47:39.307 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.307 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.307 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.307 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.307 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.307 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.308 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.308 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-26T06:47:39.308 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.308 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.308 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.308 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.308 INFO:teuthology.orchestra.run.smithi038.stdout: "contact": { 2024-03-26T06:47:39.308 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "contact", 2024-03-26T06:47:39.308 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.309 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.309 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.309 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.309 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.309 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.309 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.309 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.309 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.310 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.310 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.310 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.310 INFO:teuthology.orchestra.run.smithi038.stdout: "description": { 2024-03-26T06:47:39.310 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "description", 2024-03-26T06:47:39.310 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.310 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.310 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.311 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.311 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.311 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.311 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.311 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.311 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.311 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.311 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.311 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.312 INFO:teuthology.orchestra.run.smithi038.stdout: "device_url": { 2024-03-26T06:47:39.312 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "device_url", 2024-03-26T06:47:39.312 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.312 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.312 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.312 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-26T06:47:39.312 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.312 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.312 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.313 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.313 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.313 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.313 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.313 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.313 INFO:teuthology.orchestra.run.smithi038.stdout: "enabled": { 2024-03-26T06:47:39.313 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "enabled", 2024-03-26T06:47:39.313 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.313 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.314 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.314 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.314 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.314 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.314 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.314 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.314 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.314 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.315 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.315 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.315 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:39.315 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:39.315 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.315 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.315 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.315 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "24", 2024-03-26T06:47:39.315 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "8", 2024-03-26T06:47:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.316 INFO:teuthology.orchestra.run.smithi038.stdout: "last_opt_revision": { 2024-03-26T06:47:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "last_opt_revision", 2024-03-26T06:47:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "1", 2024-03-26T06:47:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.317 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard": { 2024-03-26T06:47:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard", 2024-03-26T06:47:39.318 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.319 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.319 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.319 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.319 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.319 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.319 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.319 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.319 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.319 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: "leaderboard_description": { 2024-03-26T06:47:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "leaderboard_description", 2024-03-26T06:47:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.320 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.321 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.321 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.321 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.321 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.321 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.321 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.321 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.321 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.321 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.322 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.322 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.322 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.322 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.322 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.322 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.322 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.322 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.323 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.323 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.323 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.323 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.323 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.323 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.323 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.323 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.323 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.324 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.324 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.324 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.324 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.324 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.324 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.324 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.324 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.325 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.325 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.325 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.325 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.325 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.325 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.325 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.325 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.325 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.326 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.326 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.326 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.326 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.326 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.326 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.326 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.326 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.327 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.327 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.327 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.327 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.327 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.327 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.327 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.327 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.328 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.328 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.328 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.328 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.328 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.328 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.328 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.328 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.329 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.329 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.329 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.329 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.329 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.329 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.329 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.329 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.329 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.330 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.330 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.330 INFO:teuthology.orchestra.run.smithi038.stdout: "organization": { 2024-03-26T06:47:39.330 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "organization", 2024-03-26T06:47:39.330 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.330 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.330 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.330 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.331 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.331 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.331 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.331 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.331 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.331 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.331 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.331 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.331 INFO:teuthology.orchestra.run.smithi038.stdout: "proxy": { 2024-03-26T06:47:39.332 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "proxy", 2024-03-26T06:47:39.332 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.332 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.332 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.332 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.332 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.332 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.332 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.332 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.333 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.333 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.333 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.333 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.333 INFO:teuthology.orchestra.run.smithi038.stdout: "url": { 2024-03-26T06:47:39.333 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "url", 2024-03-26T06:47:39.333 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.333 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.333 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.334 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-26T06:47:39.334 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.334 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.334 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.334 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.334 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.334 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.334 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.335 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.335 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.335 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.335 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.335 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "test_orchestrator", 2024-03-26T06:47:39.335 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.335 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.335 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.336 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.336 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.336 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.336 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.336 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.336 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.336 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.336 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.336 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.337 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.337 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.337 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.337 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.337 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.337 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.337 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.337 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.338 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.338 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.338 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.338 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.338 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.338 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.338 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.338 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.339 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.339 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.339 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.339 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.339 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.339 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.339 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.339 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.339 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.340 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.340 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.340 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.340 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.340 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.340 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.340 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.340 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.341 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.341 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.341 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.341 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.341 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.341 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.341 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.341 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.341 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.342 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.343 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.343 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.343 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.343 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.343 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.343 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.343 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.343 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.343 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.344 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.344 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.344 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.344 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.344 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.344 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-03-26T06:47:39.344 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.344 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.345 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.345 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.345 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.345 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.345 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.345 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.345 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.345 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.346 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.347 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.348 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.349 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.350 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.351 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.352 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "max_concurrent_clones": { 2024-03-26T06:47:39.353 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "max_concurrent_clones", 2024-03-26T06:47:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "4", 2024-03-26T06:47:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-26T06:47:39.354 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.355 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.355 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.355 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.355 INFO:teuthology.orchestra.run.smithi038.stdout: "periodic_async_work": { 2024-03-26T06:47:39.355 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "periodic_async_work", 2024-03-26T06:47:39.355 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.355 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.355 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.356 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.356 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.356 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.356 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.356 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Periodically check for async work", 2024-03-26T06:47:39.356 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.356 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.356 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.356 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.357 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_delay": { 2024-03-26T06:47:39.357 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_delay", 2024-03-26T06:47:39.357 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.357 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.357 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.357 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "0", 2024-03-26T06:47:39.357 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.357 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.358 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.358 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-26T06:47:39.358 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.358 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.358 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.358 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.358 INFO:teuthology.orchestra.run.smithi038.stdout: "snapshot_clone_no_wait": { 2024-03-26T06:47:39.358 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snapshot_clone_no_wait", 2024-03-26T06:47:39.359 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.359 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.359 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.359 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "True", 2024-03-26T06:47:39.359 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.359 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.359 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.359 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-26T06:47:39.359 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.360 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.360 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.360 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.360 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.360 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.360 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix", 2024-03-26T06:47:39.360 INFO:teuthology.orchestra.run.smithi038.stdout: "can_run": true, 2024-03-26T06:47:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "error_string": "", 2024-03-26T06:47:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "module_options": { 2024-03-26T06:47:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "discovery_interval": { 2024-03-26T06:47:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "discovery_interval", 2024-03-26T06:47:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "uint", 2024-03-26T06:47:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.361 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "100", 2024-03-26T06:47:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.362 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "identifier": { 2024-03-26T06:47:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "identifier", 2024-03-26T06:47:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.363 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "interval": { 2024-03-26T06:47:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "interval", 2024-03-26T06:47:39.364 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "secs", 2024-03-26T06:47:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "60", 2024-03-26T06:47:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.365 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: "log_level": { 2024-03-26T06:47:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_level", 2024-03-26T06:47:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.366 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.367 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.367 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.367 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.367 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.367 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.367 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.367 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.367 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.368 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.368 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.368 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.368 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.368 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.368 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.368 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.368 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster": { 2024-03-26T06:47:39.368 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster", 2024-03-26T06:47:39.369 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.369 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.369 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.369 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.369 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.369 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.369 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.369 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_cluster_level": { 2024-03-26T06:47:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_cluster_level", 2024-03-26T06:47:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.370 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "info", 2024-03-26T06:47:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [ 2024-03-26T06:47:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: "", 2024-03-26T06:47:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: "critical", 2024-03-26T06:47:39.371 INFO:teuthology.orchestra.run.smithi038.stdout: "debug", 2024-03-26T06:47:39.372 INFO:teuthology.orchestra.run.smithi038.stdout: "error", 2024-03-26T06:47:39.372 INFO:teuthology.orchestra.run.smithi038.stdout: "info", 2024-03-26T06:47:39.372 INFO:teuthology.orchestra.run.smithi038.stdout: "warning" 2024-03-26T06:47:39.372 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.372 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.372 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.372 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.372 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.373 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.373 INFO:teuthology.orchestra.run.smithi038.stdout: "log_to_file": { 2024-03-26T06:47:39.373 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "log_to_file", 2024-03-26T06:47:39.373 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "bool", 2024-03-26T06:47:39.373 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.373 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 1, 2024-03-26T06:47:39.373 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "False", 2024-03-26T06:47:39.373 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.373 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.374 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.374 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.374 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.374 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.374 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.374 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.374 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_host": { 2024-03-26T06:47:39.374 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_host", 2024-03-26T06:47:39.375 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.375 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.375 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.375 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "", 2024-03-26T06:47:39.375 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.375 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.375 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.375 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.375 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.376 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.376 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.376 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.376 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_port": { 2024-03-26T06:47:39.376 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_port", 2024-03-26T06:47:39.376 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "int", 2024-03-26T06:47:39.376 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.376 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.377 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "10051", 2024-03-26T06:47:39.377 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.377 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.377 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.377 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.377 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.377 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.377 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.378 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.378 INFO:teuthology.orchestra.run.smithi038.stdout: "zabbix_sender": { 2024-03-26T06:47:39.378 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "zabbix_sender", 2024-03-26T06:47:39.378 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "str", 2024-03-26T06:47:39.378 INFO:teuthology.orchestra.run.smithi038.stdout: "level": "advanced", 2024-03-26T06:47:39.378 INFO:teuthology.orchestra.run.smithi038.stdout: "flags": 0, 2024-03-26T06:47:39.378 INFO:teuthology.orchestra.run.smithi038.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-26T06:47:39.378 INFO:teuthology.orchestra.run.smithi038.stdout: "min": "", 2024-03-26T06:47:39.379 INFO:teuthology.orchestra.run.smithi038.stdout: "max": "", 2024-03-26T06:47:39.379 INFO:teuthology.orchestra.run.smithi038.stdout: "enum_allowed": [], 2024-03-26T06:47:39.379 INFO:teuthology.orchestra.run.smithi038.stdout: "desc": "", 2024-03-26T06:47:39.379 INFO:teuthology.orchestra.run.smithi038.stdout: "long_desc": "", 2024-03-26T06:47:39.379 INFO:teuthology.orchestra.run.smithi038.stdout: "tags": [], 2024-03-26T06:47:39.379 INFO:teuthology.orchestra.run.smithi038.stdout: "see_also": [] 2024-03-26T06:47:39.379 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.379 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.380 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.380 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.380 INFO:teuthology.orchestra.run.smithi038.stdout: "services": { 2024-03-26T06:47:39.380 INFO:teuthology.orchestra.run.smithi038.stdout: "dashboard": "https://172.21.15.38:8443/" 2024-03-26T06:47:39.380 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.380 INFO:teuthology.orchestra.run.smithi038.stdout: "always_on_modules": { 2024-03-26T06:47:39.380 INFO:teuthology.orchestra.run.smithi038.stdout: "octopus": [ 2024-03-26T06:47:39.380 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-03-26T06:47:39.381 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-03-26T06:47:39.381 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-03-26T06:47:39.381 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-03-26T06:47:39.381 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-03-26T06:47:39.381 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-03-26T06:47:39.381 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-03-26T06:47:39.381 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-03-26T06:47:39.381 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-03-26T06:47:39.381 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-03-26T06:47:39.382 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.382 INFO:teuthology.orchestra.run.smithi038.stdout: "pacific": [ 2024-03-26T06:47:39.382 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-03-26T06:47:39.382 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-03-26T06:47:39.382 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-03-26T06:47:39.382 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-03-26T06:47:39.382 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-03-26T06:47:39.382 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-03-26T06:47:39.383 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-03-26T06:47:39.383 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-03-26T06:47:39.383 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-03-26T06:47:39.383 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-03-26T06:47:39.383 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.383 INFO:teuthology.orchestra.run.smithi038.stdout: "quincy": [ 2024-03-26T06:47:39.383 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-03-26T06:47:39.383 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-03-26T06:47:39.384 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-03-26T06:47:39.384 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-03-26T06:47:39.384 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-03-26T06:47:39.384 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-03-26T06:47:39.384 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-03-26T06:47:39.384 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-03-26T06:47:39.384 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-03-26T06:47:39.384 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-03-26T06:47:39.385 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.385 INFO:teuthology.orchestra.run.smithi038.stdout: "reef": [ 2024-03-26T06:47:39.385 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-03-26T06:47:39.385 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-03-26T06:47:39.385 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-03-26T06:47:39.385 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-03-26T06:47:39.385 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-03-26T06:47:39.385 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-03-26T06:47:39.385 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-03-26T06:47:39.386 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-03-26T06:47:39.386 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-03-26T06:47:39.386 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-03-26T06:47:39.386 INFO:teuthology.orchestra.run.smithi038.stdout: ], 2024-03-26T06:47:39.386 INFO:teuthology.orchestra.run.smithi038.stdout: "squid": [ 2024-03-26T06:47:39.386 INFO:teuthology.orchestra.run.smithi038.stdout: "balancer", 2024-03-26T06:47:39.386 INFO:teuthology.orchestra.run.smithi038.stdout: "crash", 2024-03-26T06:47:39.386 INFO:teuthology.orchestra.run.smithi038.stdout: "devicehealth", 2024-03-26T06:47:39.387 INFO:teuthology.orchestra.run.smithi038.stdout: "orchestrator", 2024-03-26T06:47:39.387 INFO:teuthology.orchestra.run.smithi038.stdout: "pg_autoscaler", 2024-03-26T06:47:39.387 INFO:teuthology.orchestra.run.smithi038.stdout: "progress", 2024-03-26T06:47:39.387 INFO:teuthology.orchestra.run.smithi038.stdout: "rbd_support", 2024-03-26T06:47:39.387 INFO:teuthology.orchestra.run.smithi038.stdout: "status", 2024-03-26T06:47:39.387 INFO:teuthology.orchestra.run.smithi038.stdout: "telemetry", 2024-03-26T06:47:39.387 INFO:teuthology.orchestra.run.smithi038.stdout: "volumes" 2024-03-26T06:47:39.387 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:47:39.388 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.388 INFO:teuthology.orchestra.run.smithi038.stdout: "last_failure_osd_epoch": 89, 2024-03-26T06:47:39.388 INFO:teuthology.orchestra.run.smithi038.stdout: "active_clients": [ 2024-03-26T06:47:39.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.388 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "devicehealth", 2024-03-26T06:47:39.388 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-26T06:47:39.388 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.388 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-26T06:47:39.389 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-03-26T06:47:39.389 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 4086035472 2024-03-26T06:47:39.389 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.389 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:47:39.389 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.389 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.389 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "libcephsqlite", 2024-03-26T06:47:39.389 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-26T06:47:39.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.390 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-26T06:47:39.390 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-03-26T06:47:39.390 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1188996479 2024-03-26T06:47:39.390 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.390 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:47:39.390 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.390 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.391 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "rbd_support", 2024-03-26T06:47:39.391 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-26T06:47:39.391 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.391 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-26T06:47:39.391 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-03-26T06:47:39.391 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1041356014 2024-03-26T06:47:39.391 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.391 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:47:39.391 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.392 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-03-26T06:47:39.392 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-26T06:47:39.392 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.392 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-26T06:47:39.392 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-03-26T06:47:39.392 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1065933881 2024-03-26T06:47:39.392 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.393 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:47:39.393 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.393 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "snap_schedule", 2024-03-26T06:47:39.393 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-26T06:47:39.393 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.393 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-26T06:47:39.393 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-03-26T06:47:39.393 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3604240350 2024-03-26T06:47:39.394 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.394 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:47:39.394 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:39.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.394 INFO:teuthology.orchestra.run.smithi038.stdout: "name": "volumes", 2024-03-26T06:47:39.394 INFO:teuthology.orchestra.run.smithi038.stdout: "addrvec": [ 2024-03-26T06:47:39.394 INFO:teuthology.orchestra.run.smithi038.stdout: { 2024-03-26T06:47:39.394 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v2", 2024-03-26T06:47:39.395 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "172.21.15.38:0", 2024-03-26T06:47:39.395 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 1074111339 2024-03-26T06:47:39.395 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.395 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:47:39.395 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:39.395 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2024-03-26T06:47:39.395 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:47:39.402 DEBUG:tasks.check_counter:Getting stats from a 2024-03-26T06:47:39.402 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-26T06:47:39.403 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:37 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/66972138' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:47:39.403 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:37 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3249150555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:47:39.403 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:37 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/134569495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:47:39.403 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:38 smithi038 ceph-mon[32265]: pgmap v314: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.1 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:39.403 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:38 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1047302382' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-26T06:47:39.657 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:47:40.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:40 smithi038 ceph-mon[32265]: pgmap v315: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:40.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:40 smithi121 ceph-mon[36349]: pgmap v315: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:40.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:40 smithi170 ceph-mon[36912]: pgmap v315: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:41.447 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:47:41.448 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:47:41.448 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 215, 2024-03-26T06:47:41.448 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 202, 2024-03-26T06:47:41.448 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1574880, 2024-03-26T06:47:41.449 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 120240, 2024-03-26T06:47:41.449 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-03-26T06:47:41.449 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-03-26T06:47:41.449 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.042865119, 2024-03-26T06:47:41.449 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.012073863, 2024-03-26T06:47:41.449 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.035269444, 2024-03-26T06:47:41.449 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.004745515, 2024-03-26T06:47:41.449 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:41.450 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 202, 2024-03-26T06:47:41.450 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.009904772, 2024-03-26T06:47:41.450 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000049033 2024-03-26T06:47:41.450 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:41.450 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:41.450 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:41.450 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:41.450 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:41.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:41.451 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1574880, 2024-03-26T06:47:41.451 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 120240 2024-03-26T06:47:41.451 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:41.451 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:47:41.451 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 2, 2024-03-26T06:47:41.451 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 153, 2024-03-26T06:47:41.451 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 320, 2024-03-26T06:47:41.451 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 182016, 2024-03-26T06:47:41.452 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-03-26T06:47:41.452 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-03-26T06:47:41.452 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.013189370, 2024-03-26T06:47:41.452 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.008634865, 2024-03-26T06:47:41.452 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.002008830, 2024-03-26T06:47:41.452 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:47:41.452 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:41.452 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 150, 2024-03-26T06:47:41.453 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.010232708, 2024-03-26T06:47:41.453 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000068218 2024-03-26T06:47:41.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:41.453 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:41.453 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:41.453 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:41.453 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:41.453 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:41.453 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-26T06:47:41.454 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 182016 2024-03-26T06:47:41.454 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:41.454 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:47:41.454 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 0, 2024-03-26T06:47:41.454 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2, 2024-03-26T06:47:41.454 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 0, 2024-03-26T06:47:41.454 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 0, 2024-03-26T06:47:41.454 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-03-26T06:47:41.454 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-03-26T06:47:41.455 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.000262542, 2024-03-26T06:47:41.455 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000000000, 2024-03-26T06:47:41.455 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000205291, 2024-03-26T06:47:41.455 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:47:41.455 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:41.455 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:41.455 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:41.455 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:41.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:41.456 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:41.456 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:41.456 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:41.456 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:41.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:41.456 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-26T06:47:41.456 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-03-26T06:47:41.456 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:41.457 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-03-26T06:47:41.457 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-03-26T06:47:41.457 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-03-26T06:47:41.457 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:41.457 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-03-26T06:47:41.457 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-03-26T06:47:41.457 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-03-26T06:47:41.457 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:47:41.458 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:47:41.458 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:47:41.458 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:47:41.458 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:47:41.458 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:47:41.458 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:47:41.458 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:47:41.458 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:47:41.459 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:47:41.459 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:47:41.459 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:47:41.459 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:47:41.459 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-03-26T06:47:41.459 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:47:41.459 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-03-26T06:47:41.459 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:47:41.459 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:47:41.460 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:47:41.460 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:47:41.460 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:47:41.460 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:47:41.460 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:47:41.460 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-03-26T06:47:41.460 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:47:41.460 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:47:41.461 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:47:41.461 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-03-26T06:47:41.461 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-03-26T06:47:41.461 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-03-26T06:47:41.461 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:47:41.461 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:47:41.461 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:47:41.461 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:47:41.461 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 148977, 2024-03-26T06:47:41.462 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 30, 2024-03-26T06:47:41.462 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-03-26T06:47:41.462 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-03-26T06:47:41.462 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-03-26T06:47:41.462 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-03-26T06:47:41.462 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:47:41.462 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-03-26T06:47:41.462 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:47:41.462 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-03-26T06:47:41.463 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-03-26T06:47:41.463 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-03-26T06:47:41.463 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:47:41.463 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:47:41.463 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-03-26T06:47:41.463 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-03-26T06:47:41.463 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-03-26T06:47:41.463 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-03-26T06:47:41.464 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-03-26T06:47:41.464 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-03-26T06:47:41.464 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-03-26T06:47:41.464 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-03-26T06:47:41.464 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:41.464 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:47:41.464 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-03-26T06:47:41.464 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-03-26T06:47:41.464 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-03-26T06:47:41.465 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 213, 2024-03-26T06:47:41.465 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1542148, 2024-03-26T06:47:41.465 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:41.465 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 213, 2024-03-26T06:47:41.465 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-03-26T06:47:41.465 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-03-26T06:47:41.465 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 213, 2024-03-26T06:47:41.465 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1542148, 2024-03-26T06:47:41.465 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-03-26T06:47:41.466 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:41.466 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:41.466 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:41.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:41.466 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:41.466 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:47:42.120 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-03-26T06:47:42.120 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-03-26T06:47:42.120 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-26T06:47:42.121 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-26T06:47:42.121 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-26T06:47:42.121 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-03-26T06:47:42.121 DEBUG:tasks.check_counter:Getting stats from d 2024-03-26T06:47:42.121 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-26T06:47:42.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:42 smithi121 ceph-mon[36349]: pgmap v316: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:42.408 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:47:42.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:42 smithi170 ceph-mon[36912]: pgmap v316: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:42.431 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:42 smithi038 ceph-mon[32265]: pgmap v316: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:44.231 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:44 smithi038 ceph-mon[32265]: pgmap v317: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T06:47:44.291 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:47:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:47:44.291 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 221, 2024-03-26T06:47:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 208, 2024-03-26T06:47:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1576512, 2024-03-26T06:47:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 123984, 2024-03-26T06:47:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-03-26T06:47:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-03-26T06:47:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.043690376, 2024-03-26T06:47:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.013555249, 2024-03-26T06:47:44.292 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.034590245, 2024-03-26T06:47:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.004455354, 2024-03-26T06:47:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 208, 2024-03-26T06:47:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.026255154, 2024-03-26T06:47:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000126226 2024-03-26T06:47:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:44.293 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1576512, 2024-03-26T06:47:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 123984 2024-03-26T06:47:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:47:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 2, 2024-03-26T06:47:44.294 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 158, 2024-03-26T06:47:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 320, 2024-03-26T06:47:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 182896, 2024-03-26T06:47:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-03-26T06:47:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-03-26T06:47:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.013421734, 2024-03-26T06:47:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.008840373, 2024-03-26T06:47:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.001940078, 2024-03-26T06:47:44.295 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:47:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 155, 2024-03-26T06:47:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.010520228, 2024-03-26T06:47:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000067872 2024-03-26T06:47:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:44.296 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-26T06:47:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 182896 2024-03-26T06:47:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:47:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 0, 2024-03-26T06:47:44.297 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2, 2024-03-26T06:47:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 0, 2024-03-26T06:47:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 0, 2024-03-26T06:47:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-03-26T06:47:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-03-26T06:47:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.000356114, 2024-03-26T06:47:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000000000, 2024-03-26T06:47:44.298 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000299455, 2024-03-26T06:47:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:47:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:44.299 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-26T06:47:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-03-26T06:47:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-03-26T06:47:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-03-26T06:47:44.300 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-03-26T06:47:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-03-26T06:47:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-03-26T06:47:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-03-26T06:47:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:47:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:47:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:47:44.301 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:47:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:47:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:47:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:47:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:47:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:47:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:47:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:47:44.302 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:47:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:47:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-03-26T06:47:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:47:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-03-26T06:47:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:47:44.303 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:47:44.304 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:47:44.304 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:47:44.304 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:47:44.304 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:47:44.304 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:47:44.304 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-03-26T06:47:44.304 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:47:44.304 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:47:44.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:47:44.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-03-26T06:47:44.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-03-26T06:47:44.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-03-26T06:47:44.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:47:44.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:47:44.305 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:47:44.306 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:47:44.306 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 149057, 2024-03-26T06:47:44.306 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 32, 2024-03-26T06:47:44.306 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-03-26T06:47:44.306 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-03-26T06:47:44.307 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-03-26T06:47:44.307 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-03-26T06:47:44.307 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:47:44.307 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-03-26T06:47:44.307 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:47:44.307 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-03-26T06:47:44.307 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-03-26T06:47:44.308 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-03-26T06:47:44.308 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:47:44.308 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:47:44.308 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-03-26T06:47:44.308 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-03-26T06:47:44.308 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-03-26T06:47:44.308 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-03-26T06:47:44.308 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-03-26T06:47:44.309 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-03-26T06:47:44.309 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-03-26T06:47:44.309 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-03-26T06:47:44.309 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:44.309 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:47:44.309 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-03-26T06:47:44.309 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-03-26T06:47:44.310 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-03-26T06:47:44.310 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 219, 2024-03-26T06:47:44.310 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1542928, 2024-03-26T06:47:44.310 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:44.310 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 219, 2024-03-26T06:47:44.310 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-03-26T06:47:44.311 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-03-26T06:47:44.311 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 219, 2024-03-26T06:47:44.311 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1542928, 2024-03-26T06:47:44.311 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-03-26T06:47:44.311 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:44.311 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:44.311 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:44.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:44.312 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:44.312 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:47:44.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:44 smithi121 ceph-mon[36349]: pgmap v317: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T06:47:44.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:44 smithi170 ceph-mon[36912]: pgmap v317: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-26T06:47:44.931 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-26T06:47:44.931 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-26T06:47:44.931 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-26T06:47:44.931 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-26T06:47:44.932 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-26T06:47:44.932 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-26T06:47:44.932 DEBUG:tasks.check_counter:Getting stats from g 2024-03-26T06:47:44.932 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-26T06:47:45.181 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:47:46.385 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:46 smithi038 ceph-mon[32265]: pgmap v318: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:46.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:46 smithi121 ceph-mon[36349]: pgmap v318: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:46.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:46 smithi170 ceph-mon[36912]: pgmap v318: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:46.981 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:47:46.981 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:47:46.981 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 4, 2024-03-26T06:47:46.981 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 4, 2024-03-26T06:47:46.982 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 3024, 2024-03-26T06:47:46.982 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 448, 2024-03-26T06:47:46.982 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-03-26T06:47:46.982 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-03-26T06:47:46.982 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.001582481, 2024-03-26T06:47:46.982 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000041270, 2024-03-26T06:47:46.983 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000629217, 2024-03-26T06:47:46.983 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:47:46.983 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:46.983 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 3, 2024-03-26T06:47:46.983 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000782296, 2024-03-26T06:47:46.983 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000260765 2024-03-26T06:47:46.983 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:46.984 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:46.984 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:46.984 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:46.984 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:46.984 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:46.984 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-26T06:47:46.984 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 448 2024-03-26T06:47:46.985 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:46.985 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:47:46.985 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 2, 2024-03-26T06:47:46.985 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 154, 2024-03-26T06:47:46.985 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 320, 2024-03-26T06:47:46.985 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 182192, 2024-03-26T06:47:46.986 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-03-26T06:47:46.986 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-03-26T06:47:46.986 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.011874872, 2024-03-26T06:47:46.986 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.008169322, 2024-03-26T06:47:46.986 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.001906866, 2024-03-26T06:47:46.986 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:47:46.987 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:46.987 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 151, 2024-03-26T06:47:46.987 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.020825981, 2024-03-26T06:47:46.987 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000137920 2024-03-26T06:47:46.987 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:46.987 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:46.988 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:46.988 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:46.988 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:46.988 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:46.988 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-26T06:47:46.989 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 182192 2024-03-26T06:47:46.989 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:46.989 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:47:46.989 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 213, 2024-03-26T06:47:46.989 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 202, 2024-03-26T06:47:46.989 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1572400, 2024-03-26T06:47:46.989 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 121040, 2024-03-26T06:47:46.990 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-03-26T06:47:46.990 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-03-26T06:47:46.990 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.040197106, 2024-03-26T06:47:46.990 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.012071270, 2024-03-26T06:47:46.990 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.031636374, 2024-03-26T06:47:46.990 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.005673262, 2024-03-26T06:47:46.990 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:46.991 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 201, 2024-03-26T06:47:46.991 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.009029264, 2024-03-26T06:47:46.991 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000044921 2024-03-26T06:47:46.991 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:46.991 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:46.991 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:46.991 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:46.992 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:46.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:46.992 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1572400, 2024-03-26T06:47:46.992 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 121040 2024-03-26T06:47:46.992 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:46.992 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-03-26T06:47:46.993 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-03-26T06:47:46.993 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-03-26T06:47:46.993 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:46.993 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-03-26T06:47:46.993 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-03-26T06:47:46.993 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-03-26T06:47:46.993 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:47:46.994 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:47:46.994 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:47:46.994 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:47:46.994 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:47:46.994 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:47:46.994 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:47:46.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:47:46.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:47:46.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:47:46.995 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:47:46.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:47:46.996 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:47:46.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-03-26T06:47:46.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:47:46.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-03-26T06:47:46.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:47:46.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:47:46.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:47:46.997 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:47:46.998 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:47:46.998 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:47:46.998 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:47:46.998 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-03-26T06:47:46.998 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:47:46.998 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:47:46.998 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:47:46.998 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-03-26T06:47:46.999 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-03-26T06:47:46.999 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-03-26T06:47:46.999 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:47:46.999 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:47:46.999 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:47:46.999 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:47:46.999 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 148977, 2024-03-26T06:47:47.000 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 30, 2024-03-26T06:47:47.000 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-03-26T06:47:47.000 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-03-26T06:47:47.000 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-03-26T06:47:47.000 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-03-26T06:47:47.000 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:47:47.000 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-03-26T06:47:47.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:47:47.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-03-26T06:47:47.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-03-26T06:47:47.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-03-26T06:47:47.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:47:47.001 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:47:47.001 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-03-26T06:47:47.001 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-03-26T06:47:47.001 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-03-26T06:47:47.002 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-03-26T06:47:47.002 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-03-26T06:47:47.002 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-03-26T06:47:47.002 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-03-26T06:47:47.002 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-03-26T06:47:47.002 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:47.002 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:47:47.002 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-03-26T06:47:47.003 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-03-26T06:47:47.003 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-03-26T06:47:47.003 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 215, 2024-03-26T06:47:47.003 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1542408, 2024-03-26T06:47:47.003 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:47.003 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 215, 2024-03-26T06:47:47.003 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-03-26T06:47:47.003 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-03-26T06:47:47.004 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 215, 2024-03-26T06:47:47.004 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1542408, 2024-03-26T06:47:47.004 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-03-26T06:47:47.004 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:47.004 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:47.004 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:47.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:47.004 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:47.004 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:47:47.703 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-26T06:47:47.703 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-26T06:47:47.704 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-26T06:47:47.704 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-26T06:47:47.704 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-26T06:47:47.704 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-26T06:47:47.704 DEBUG:tasks.check_counter:Getting stats from j 2024-03-26T06:47:47.704 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-26T06:47:47.970 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T06:47:48.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:48 smithi121 ceph-mon[36349]: pgmap v319: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-26T06:47:48.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:48 smithi170 ceph-mon[36912]: pgmap v319: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-26T06:47:48.422 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:48 smithi038 ceph-mon[32265]: pgmap v319: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 8 op/s 2024-03-26T06:47:49.372 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:49 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:47:49.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:49 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:47:49.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:49 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T06:47:49.778 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2024-03-26T06:47:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:47:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 0, 2024-03-26T06:47:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 2, 2024-03-26T06:47:49.779 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 0, 2024-03-26T06:47:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 0, 2024-03-26T06:47:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 4, 2024-03-26T06:47:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 0, 2024-03-26T06:47:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.001357574, 2024-03-26T06:47:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.000000000, 2024-03-26T06:47:49.780 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.000354528, 2024-03-26T06:47:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:47:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:49.781 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-26T06:47:49.782 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 0 2024-03-26T06:47:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:47:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 6, 2024-03-26T06:47:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 161, 2024-03-26T06:47:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 3344, 2024-03-26T06:47:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 183520, 2024-03-26T06:47:49.783 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 5, 2024-03-26T06:47:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 3, 2024-03-26T06:47:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.013345096, 2024-03-26T06:47:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.008771163, 2024-03-26T06:47:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.002073335, 2024-03-26T06:47:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:47:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:49.784 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 159, 2024-03-26T06:47:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.019450396, 2024-03-26T06:47:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000122329 2024-03-26T06:47:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:49.785 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-26T06:47:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 183520 2024-03-26T06:47:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:47:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_messages": 219, 2024-03-26T06:47:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages": 208, 2024-03-26T06:47:49.786 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_bytes": 1574032, 2024-03-26T06:47:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_bytes": 124784, 2024-03-26T06:47:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_created_connections": 2, 2024-03-26T06:47:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_active_connections": 1, 2024-03-26T06:47:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_total_time": 0.041453168, 2024-03-26T06:47:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_send_time": 0.013381084, 2024-03-26T06:47:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_recv_time": 0.033128428, 2024-03-26T06:47:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_running_fast_dispatch_time": 0.004481687, 2024-03-26T06:47:49.787 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 207, 2024-03-26T06:47:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.012594098, 2024-03-26T06:47:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000060841 2024-03-26T06:47:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:49.788 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_recv_encrypted_bytes": 1574032, 2024-03-26T06:47:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "msgr_send_encrypted_bytes": 124784 2024-03-26T06:47:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "cct": { 2024-03-26T06:47:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "total_workers": 0, 2024-03-26T06:47:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "unhealthy_workers": 0 2024-03-26T06:47:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "mempool": { 2024-03-26T06:47:49.789 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_bytes": 0, 2024-03-26T06:47:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "bloom_filter_items": 0, 2024-03-26T06:47:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:47:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:47:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:47:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:47:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:47:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:47:49.790 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:47:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:47:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:47:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:47:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:47:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:47:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:47:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_extent_items": 0, 2024-03-26T06:47:49.791 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:47:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_blob_items": 0, 2024-03-26T06:47:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:47:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:47:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:47:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:47:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:47:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:47:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:47:49.792 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_txc_items": 0, 2024-03-26T06:47:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:47:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:47:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:47:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "bluestore_writing_items": 0, 2024-03-26T06:47:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_bytes": 0, 2024-03-26T06:47:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_items": 0, 2024-03-26T06:47:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:47:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:47:49.793 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:47:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:47:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_bytes": 149034, 2024-03-26T06:47:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_anon_items": 32, 2024-03-26T06:47:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_bytes": 0, 2024-03-26T06:47:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "buffer_meta_items": 0, 2024-03-26T06:47:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_bytes": 0, 2024-03-26T06:47:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_items": 0, 2024-03-26T06:47:49.794 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:47:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_mapbl_items": 0, 2024-03-26T06:47:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:47:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_pglog_items": 0, 2024-03-26T06:47:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_bytes": 0, 2024-03-26T06:47:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_items": 0, 2024-03-26T06:47:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:47:49.795 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:47:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_bytes": 0, 2024-03-26T06:47:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "pgmap_items": 0, 2024-03-26T06:47:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_bytes": 928, 2024-03-26T06:47:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_co_items": 2, 2024-03-26T06:47:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_bytes": 0, 2024-03-26T06:47:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_1_items": 0, 2024-03-26T06:47:49.796 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_bytes": 0, 2024-03-26T06:47:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "unittest_2_items": 0 2024-03-26T06:47:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2024-03-26T06:47:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:47:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "val": 0, 2024-03-26T06:47:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "max": 104857600, 2024-03-26T06:47:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "get_started": 0, 2024-03-26T06:47:49.797 INFO:teuthology.orchestra.run.smithi038.stdout: "get": 221, 2024-03-26T06:47:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "get_sum": 1543188, 2024-03-26T06:47:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "get_or_fail_success": 221, 2024-03-26T06:47:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "take": 0, 2024-03-26T06:47:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "take_sum": 0, 2024-03-26T06:47:49.798 INFO:teuthology.orchestra.run.smithi038.stdout: "put": 221, 2024-03-26T06:47:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "put_sum": 1543188, 2024-03-26T06:47:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "wait": { 2024-03-26T06:47:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "avgcount": 0, 2024-03-26T06:47:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "sum": 0.000000000, 2024-03-26T06:47:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: "avgtime": 0.000000000 2024-03-26T06:47:49.799 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:49.800 INFO:teuthology.orchestra.run.smithi038.stdout: } 2024-03-26T06:47:49.800 INFO:teuthology.orchestra.run.smithi038.stdout:} 2024-03-26T06:47:50.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:50 smithi121 ceph-mon[36349]: pgmap v320: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:50.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:50 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:50 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:50 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:50 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:50 smithi170 ceph-mon[36912]: pgmap v320: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:50.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:50 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:50 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:50 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:50 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.470 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:50 smithi038 ceph-mon[32265]: pgmap v320: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T06:47:50.470 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:50 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.470 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:50 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.471 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:50 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.471 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:50 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:50.497 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-26T06:47:50.498 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-26T06:47:50.498 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-26T06:47:50.498 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-26T06:47:50.498 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-26T06:47:50.498 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-26T06:47:50.498 DEBUG:tasks.check_counter:Getting stats from b 2024-03-26T06:47:50.499 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-26T06:47:50.739 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:47:51.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.396 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi121 to 174.1M 2024-03-26T06:47:51.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: Unable to set osd_memory_target on smithi121 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-26T06:47:51.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.397 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-26T06:47:51.398 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:51 smithi121 ceph-mon[36349]: Unable to set osd_memory_target on smithi170 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-26T06:47:51.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.419 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.420 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi121 to 174.1M 2024-03-26T06:47:51.420 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: Unable to set osd_memory_target on smithi121 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-26T06:47:51.420 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.420 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.420 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.421 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.421 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-26T06:47:51.421 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:51 smithi170 ceph-mon[36912]: Unable to set osd_memory_target on smithi170 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-26T06:47:51.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi121 to 174.1M 2024-03-26T06:47:51.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: Unable to set osd_memory_target on smithi121 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-03-26T06:47:51.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.585 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:51.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi170 to 174.1M 2024-03-26T06:47:51.586 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:51 smithi038 ceph-mon[32265]: Unable to set osd_memory_target on smithi170 to 182638489: error parsing value: Value '182638489' is below minimum 939524096 2024-03-26T06:47:52.411 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-03-26T06:47:52.412 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:47:52.412 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 2927, 2024-03-26T06:47:52.412 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 3271, 2024-03-26T06:47:52.412 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 918451, 2024-03-26T06:47:52.412 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 40022314, 2024-03-26T06:47:52.412 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 12, 2024-03-26T06:47:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 4, 2024-03-26T06:47:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.398678674, 2024-03-26T06:47:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.218572178, 2024-03-26T06:47:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.079504596, 2024-03-26T06:47:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.085641604, 2024-03-26T06:47:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 3268, 2024-03-26T06:47:52.413 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.681405289, 2024-03-26T06:47:52.414 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000208508 2024-03-26T06:47:52.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.414 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:52.414 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2335, 2024-03-26T06:47:52.414 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000542816, 2024-03-26T06:47:52.414 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000232 2024-03-26T06:47:52.414 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.415 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 2016, 2024-03-26T06:47:52.415 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 312720 2024-03-26T06:47:52.415 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.415 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:47:52.415 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 30619, 2024-03-26T06:47:52.415 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 33530, 2024-03-26T06:47:52.415 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 11119970, 2024-03-26T06:47:52.415 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 25487931, 2024-03-26T06:47:52.416 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 14, 2024-03-26T06:47:52.416 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 6, 2024-03-26T06:47:52.416 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 1.883346649, 2024-03-26T06:47:52.416 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.673323368, 2024-03-26T06:47:52.416 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 2.005945708, 2024-03-26T06:47:52.416 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.173146191, 2024-03-26T06:47:52.416 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:52.416 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 33530, 2024-03-26T06:47:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 17.754869673, 2024-03-26T06:47:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000529521 2024-03-26T06:47:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 21579, 2024-03-26T06:47:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.013728234, 2024-03-26T06:47:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000636 2024-03-26T06:47:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.417 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 2014000, 2024-03-26T06:47:52.418 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 247936 2024-03-26T06:47:52.418 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.418 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:47:52.418 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 1759, 2024-03-26T06:47:52.418 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 1903, 2024-03-26T06:47:52.418 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 665264, 2024-03-26T06:47:52.418 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 8491911, 2024-03-26T06:47:52.418 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 46, 2024-03-26T06:47:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 9, 2024-03-26T06:47:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.227361550, 2024-03-26T06:47:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.070083785, 2024-03-26T06:47:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.070854621, 2024-03-26T06:47:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.067464961, 2024-03-26T06:47:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1900, 2024-03-26T06:47:52.419 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.625336808, 2024-03-26T06:47:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000329124 2024-03-26T06:47:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1532, 2024-03-26T06:47:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000306702, 2024-03-26T06:47:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000200 2024-03-26T06:47:52.420 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.421 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 42384, 2024-03-26T06:47:52.421 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 131472 2024-03-26T06:47:52.421 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.421 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2024-03-26T06:47:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 1, 2024-03-26T06:47:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2024-03-26T06:47:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-MDSRank": { 2024-03-26T06:47:52.422 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2024-03-26T06:47:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2024-03-26T06:47:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 4160, 2024-03-26T06:47:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 17.908512124, 2024-03-26T06:47:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.004304930 2024-03-26T06:47:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-PurgeQueue": { 2024-03-26T06:47:52.423 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2024-03-26T06:47:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2024-03-26T06:47:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 813, 2024-03-26T06:47:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.056185548, 2024-03-26T06:47:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000069108 2024-03-26T06:47:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-03-26T06:47:52.424 INFO:teuthology.orchestra.run.smithi121.stdout: "request": 1981, 2024-03-26T06:47:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: "reply": 1980, 2024-03-26T06:47:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: "reply_latency": { 2024-03-26T06:47:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1980, 2024-03-26T06:47:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 17.003986321, 2024-03-26T06:47:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.008587871 2024-03-26T06:47:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_reply": 0, 2024-03-26T06:47:52.425 INFO:teuthology.orchestra.run.smithi121.stdout: "forward": 1, 2024-03-26T06:47:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_complete": 0, 2024-03-26T06:47:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_keys": 0, 2024-03-26T06:47:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_commit": 213, 2024-03-26T06:47:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_split": 3, 2024-03-26T06:47:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_merge": 0, 2024-03-26T06:47:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes": 920, 2024-03-26T06:47:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_top": 298, 2024-03-26T06:47:52.426 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_bottom": 599, 2024-03-26T06:47:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pin_tail": 23, 2024-03-26T06:47:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pinned": 404, 2024-03-26T06:47:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_expired": 182, 2024-03-26T06:47:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_with_caps": 253, 2024-03-26T06:47:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: "caps": 253, 2024-03-26T06:47:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: "subtrees": 6, 2024-03-26T06:47:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse": 2689, 2024-03-26T06:47:52.427 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_hit": 2530, 2024-03-26T06:47:52.428 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_forward": 0, 2024-03-26T06:47:52.428 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_discover": 0, 2024-03-26T06:47:52.428 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_dir_fetch": 0, 2024-03-26T06:47:52.428 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_remote_ino": 0, 2024-03-26T06:47:52.428 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_lock": 55, 2024-03-26T06:47:52.428 INFO:teuthology.orchestra.run.smithi121.stdout: "load_cent": 0, 2024-03-26T06:47:52.428 INFO:teuthology.orchestra.run.smithi121.stdout: "q": 0, 2024-03-26T06:47:52.428 INFO:teuthology.orchestra.run.smithi121.stdout: "exported": 65, 2024-03-26T06:47:52.429 INFO:teuthology.orchestra.run.smithi121.stdout: "exported_inodes": 227, 2024-03-26T06:47:52.429 INFO:teuthology.orchestra.run.smithi121.stdout: "imported": 2, 2024-03-26T06:47:52.429 INFO:teuthology.orchestra.run.smithi121.stdout: "imported_inodes": 2, 2024-03-26T06:47:52.429 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_dir_fetch": 0, 2024-03-26T06:47:52.429 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_backtrace_fetch": 0, 2024-03-26T06:47:52.429 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_peer_discover": 0, 2024-03-26T06:47:52.429 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rfiles": 324, 2024-03-26T06:47:52.429 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rbytes": 13424234229, 2024-03-26T06:47:52.430 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rsnaps": 6, 2024-03-26T06:47:52.430 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_fetch": 0, 2024-03-26T06:47:52.430 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_set_tag": 0, 2024-03-26T06:47:52.430 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_repaired": 0, 2024-03-26T06:47:52.430 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_inotable_repaired": 0, 2024-03-26T06:47:52.430 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_inodes": 0, 2024-03-26T06:47:52.430 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_base_inodes": 0, 2024-03-26T06:47:52.430 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dirfrag_rstats": 0, 2024-03-26T06:47:52.431 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_file_inodes": 0, 2024-03-26T06:47:52.431 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_inode_file_caps": 1, 2024-03-26T06:47:52.431 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_revoke": 642, 2024-03-26T06:47:52.431 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_grant": 3444, 2024-03-26T06:47:52.431 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_trunc": 6, 2024-03-26T06:47:52.431 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-26T06:47:52.431 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-26T06:47:52.431 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps": 25082, 2024-03-26T06:47:52.431 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps_dirty": 23746, 2024-03-26T06:47:52.432 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_cap_release": 238, 2024-03-26T06:47:52.432 INFO:teuthology.orchestra.run.smithi121.stdout: "process_request_cap_release": 837 2024-03-26T06:47:52.432 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.432 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache": { 2024-03-26T06:47:52.432 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update": 0, 2024-03-26T06:47:52.432 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update_receipt": 0, 2024-03-26T06:47:52.432 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_try_discover": 0, 2024-03-26T06:47:52.432 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_send_discover": 0, 2024-03-26T06:47:52.433 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_handle_discover": 73, 2024-03-26T06:47:52.433 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays": 0, 2024-03-26T06:47:52.433 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_delayed": 0, 2024-03-26T06:47:52.433 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_enqueuing": 0, 2024-03-26T06:47:52.433 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_created": 530, 2024-03-26T06:47:52.433 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_enqueued": 469, 2024-03-26T06:47:52.433 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_reintegrated": 61, 2024-03-26T06:47:52.433 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_migrated": 0, 2024-03-26T06:47:52.433 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_processing": 0, 2024-03-26T06:47:52.434 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_enqueued": 0, 2024-03-26T06:47:52.434 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_prioritized": 0, 2024-03-26T06:47:52.434 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_started": 0, 2024-03-26T06:47:52.434 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_completed": 0, 2024-03-26T06:47:52.434 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_enqueue_scrub": 0, 2024-03-26T06:47:52.434 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_exportdir": 65, 2024-03-26T06:47:52.434 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_flush": 0, 2024-03-26T06:47:52.434 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragmentdir": 3, 2024-03-26T06:47:52.435 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragstats": 0, 2024-03-26T06:47:52.435 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_inodestats": 0 2024-03-26T06:47:52.435 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.435 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log": { 2024-03-26T06:47:52.435 INFO:teuthology.orchestra.run.smithi121.stdout: "evlrg": 0, 2024-03-26T06:47:52.435 INFO:teuthology.orchestra.run.smithi121.stdout: "evadd": 26930, 2024-03-26T06:47:52.435 INFO:teuthology.orchestra.run.smithi121.stdout: "evex": 25485, 2024-03-26T06:47:52.435 INFO:teuthology.orchestra.run.smithi121.stdout: "evtrm": 25485, 2024-03-26T06:47:52.436 INFO:teuthology.orchestra.run.smithi121.stdout: "ev": 1445, 2024-03-26T06:47:52.436 INFO:teuthology.orchestra.run.smithi121.stdout: "evexg": 0, 2024-03-26T06:47:52.436 INFO:teuthology.orchestra.run.smithi121.stdout: "evexd": 230, 2024-03-26T06:47:52.436 INFO:teuthology.orchestra.run.smithi121.stdout: "segadd": 49, 2024-03-26T06:47:52.436 INFO:teuthology.orchestra.run.smithi121.stdout: "segex": 47, 2024-03-26T06:47:52.436 INFO:teuthology.orchestra.run.smithi121.stdout: "segtrm": 47, 2024-03-26T06:47:52.436 INFO:teuthology.orchestra.run.smithi121.stdout: "seg": 2, 2024-03-26T06:47:52.436 INFO:teuthology.orchestra.run.smithi121.stdout: "segmjr": 1, 2024-03-26T06:47:52.437 INFO:teuthology.orchestra.run.smithi121.stdout: "segexg": 0, 2024-03-26T06:47:52.437 INFO:teuthology.orchestra.run.smithi121.stdout: "segexd": 1, 2024-03-26T06:47:52.437 INFO:teuthology.orchestra.run.smithi121.stdout: "expos": 65025290, 2024-03-26T06:47:52.437 INFO:teuthology.orchestra.run.smithi121.stdout: "wrpos": 70888444, 2024-03-26T06:47:52.437 INFO:teuthology.orchestra.run.smithi121.stdout: "rdpos": 0, 2024-03-26T06:47:52.437 INFO:teuthology.orchestra.run.smithi121.stdout: "jlat": { 2024-03-26T06:47:52.437 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2299, 2024-03-26T06:47:52.437 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 252.286641026, 2024-03-26T06:47:52.438 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.109737555 2024-03-26T06:47:52.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.438 INFO:teuthology.orchestra.run.smithi121.stdout: "replayed": 0 2024-03-26T06:47:52.438 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.438 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_mem": { 2024-03-26T06:47:52.438 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 831, 2024-03-26T06:47:52.438 INFO:teuthology.orchestra.run.smithi121.stdout: "ino+": 1397, 2024-03-26T06:47:52.438 INFO:teuthology.orchestra.run.smithi121.stdout: "ino-": 566, 2024-03-26T06:47:52.439 INFO:teuthology.orchestra.run.smithi121.stdout: "dir": 111, 2024-03-26T06:47:52.439 INFO:teuthology.orchestra.run.smithi121.stdout: "dir+": 141, 2024-03-26T06:47:52.439 INFO:teuthology.orchestra.run.smithi121.stdout: "dir-": 30, 2024-03-26T06:47:52.439 INFO:teuthology.orchestra.run.smithi121.stdout: "dn": 920, 2024-03-26T06:47:52.439 INFO:teuthology.orchestra.run.smithi121.stdout: "dn+": 1697, 2024-03-26T06:47:52.439 INFO:teuthology.orchestra.run.smithi121.stdout: "dn-": 777, 2024-03-26T06:47:52.439 INFO:teuthology.orchestra.run.smithi121.stdout: "cap": 253, 2024-03-26T06:47:52.439 INFO:teuthology.orchestra.run.smithi121.stdout: "cap+": 892, 2024-03-26T06:47:52.439 INFO:teuthology.orchestra.run.smithi121.stdout: "cap-": 639, 2024-03-26T06:47:52.440 INFO:teuthology.orchestra.run.smithi121.stdout: "rss": 71616, 2024-03-26T06:47:52.440 INFO:teuthology.orchestra.run.smithi121.stdout: "heap": 215420 2024-03-26T06:47:52.440 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.440 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_server": { 2024-03-26T06:47:52.440 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_client_request": 2450, 2024-03-26T06:47:52.440 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_server_request": 0, 2024-03-26T06:47:52.440 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_request": 1981, 2024-03-26T06:47:52.440 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_session": 89, 2024-03-26T06:47:52.441 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_peer_request": 1, 2024-03-26T06:47:52.441 INFO:teuthology.orchestra.run.smithi121.stdout: "req_create_latency": { 2024-03-26T06:47:52.441 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 764, 2024-03-26T06:47:52.441 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.766715208, 2024-03-26T06:47:52.441 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.001003553 2024-03-26T06:47:52.441 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.441 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getattr_latency": { 2024-03-26T06:47:52.441 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 164, 2024-03-26T06:47:52.441 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.051671918, 2024-03-26T06:47:52.442 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000315072 2024-03-26T06:47:52.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.442 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getfilelock_latency": { 2024-03-26T06:47:52.442 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.442 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.442 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.442 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.442 INFO:teuthology.orchestra.run.smithi121.stdout: "req_link_latency": { 2024-03-26T06:47:52.443 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 61, 2024-03-26T06:47:52.443 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.051421471, 2024-03-26T06:47:52.443 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000842974 2024-03-26T06:47:52.443 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.443 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookup_latency": { 2024-03-26T06:47:52.443 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 79, 2024-03-26T06:47:52.443 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.111530820, 2024-03-26T06:47:52.443 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.001411782 2024-03-26T06:47:52.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.444 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookuphash_latency": { 2024-03-26T06:47:52.444 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.444 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.444 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.444 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.444 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupino_latency": { 2024-03-26T06:47:52.444 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1, 2024-03-26T06:47:52.444 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000111113, 2024-03-26T06:47:52.445 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000111113 2024-03-26T06:47:52.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.445 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupname_latency": { 2024-03-26T06:47:52.445 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.445 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.445 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.445 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.445 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupparent_latency": { 2024-03-26T06:47:52.446 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.446 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.446 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.446 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.446 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupsnap_latency": { 2024-03-26T06:47:52.446 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.446 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.446 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.447 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.447 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lssnap_latency": { 2024-03-26T06:47:52.447 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 10, 2024-03-26T06:47:52.447 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.004509644, 2024-03-26T06:47:52.448 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000450964 2024-03-26T06:47:52.448 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.448 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mkdir_latency": { 2024-03-26T06:47:52.448 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 105, 2024-03-26T06:47:52.448 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.104630859, 2024-03-26T06:47:52.448 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000996484 2024-03-26T06:47:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mknod_latency": { 2024-03-26T06:47:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mksnap_latency": { 2024-03-26T06:47:52.449 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 10, 2024-03-26T06:47:52.450 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 1.584184467, 2024-03-26T06:47:52.450 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.158418446 2024-03-26T06:47:52.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.450 INFO:teuthology.orchestra.run.smithi121.stdout: "req_open_latency": { 2024-03-26T06:47:52.450 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 48, 2024-03-26T06:47:52.450 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.014114481, 2024-03-26T06:47:52.450 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000294051 2024-03-26T06:47:52.450 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.451 INFO:teuthology.orchestra.run.smithi121.stdout: "req_readdir_latency": { 2024-03-26T06:47:52.451 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 92, 2024-03-26T06:47:52.451 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.922660175, 2024-03-26T06:47:52.451 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.010028914 2024-03-26T06:47:52.451 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.451 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rename_latency": { 2024-03-26T06:47:52.451 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 113, 2024-03-26T06:47:52.451 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 12.407373480, 2024-03-26T06:47:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.109799765 2024-03-26T06:47:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "req_renamesnap_latency": { 2024-03-26T06:47:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.452 INFO:teuthology.orchestra.run.smithi121.stdout: "req_snapdiff_latency": { 2024-03-26T06:47:52.453 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.453 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.453 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.453 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmdir_latency": { 2024-03-26T06:47:52.453 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 16, 2024-03-26T06:47:52.453 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.144812438, 2024-03-26T06:47:52.453 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.009050777 2024-03-26T06:47:52.453 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.454 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmsnap_latency": { 2024-03-26T06:47:52.454 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 4, 2024-03-26T06:47:52.454 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.019177197, 2024-03-26T06:47:52.454 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.004794299 2024-03-26T06:47:52.454 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.454 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmxattr_latency": { 2024-03-26T06:47:52.454 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.454 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.455 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.455 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setattr_latency": { 2024-03-26T06:47:52.455 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 7, 2024-03-26T06:47:52.455 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.031080593, 2024-03-26T06:47:52.455 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.004440084 2024-03-26T06:47:52.455 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.455 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setdirlayout_latency": { 2024-03-26T06:47:52.456 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.456 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.456 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.456 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.456 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setfilelock_latency": { 2024-03-26T06:47:52.456 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 4, 2024-03-26T06:47:52.456 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000877832, 2024-03-26T06:47:52.456 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000219458 2024-03-26T06:47:52.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.457 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setlayout_latency": { 2024-03-26T06:47:52.457 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.457 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.457 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.457 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.457 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setxattr_latency": { 2024-03-26T06:47:52.457 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 8, 2024-03-26T06:47:52.457 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.020856112, 2024-03-26T06:47:52.458 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.002607014 2024-03-26T06:47:52.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.458 INFO:teuthology.orchestra.run.smithi121.stdout: "req_symlink_latency": { 2024-03-26T06:47:52.458 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 13, 2024-03-26T06:47:52.458 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.007955056, 2024-03-26T06:47:52.458 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000611927 2024-03-26T06:47:52.458 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.458 INFO:teuthology.orchestra.run.smithi121.stdout: "req_unlink_latency": { 2024-03-26T06:47:52.459 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 481, 2024-03-26T06:47:52.459 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.760303457, 2024-03-26T06:47:52.459 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.001580672 2024-03-26T06:47:52.459 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.459 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_revoke_eviction": 0, 2024-03-26T06:47:52.459 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_acquisition_throttle": 0, 2024-03-26T06:47:52.460 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getvxattr_latency": { 2024-03-26T06:47:52.460 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.460 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.460 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.460 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:52.460 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.461 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_sessions": { 2024-03-26T06:47:52.461 INFO:teuthology.orchestra.run.smithi121.stdout: "session_count": 2, 2024-03-26T06:47:52.461 INFO:teuthology.orchestra.run.smithi121.stdout: "session_add": 5, 2024-03-26T06:47:52.461 INFO:teuthology.orchestra.run.smithi121.stdout: "session_remove": 3, 2024-03-26T06:47:52.461 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_open": 2, 2024-03-26T06:47:52.462 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_stale": 0, 2024-03-26T06:47:52.462 INFO:teuthology.orchestra.run.smithi121.stdout: "total_load": 34, 2024-03-26T06:47:52.462 INFO:teuthology.orchestra.run.smithi121.stdout: "average_load": 17, 2024-03-26T06:47:52.462 INFO:teuthology.orchestra.run.smithi121.stdout: "avg_session_uptime": 610, 2024-03-26T06:47:52.462 INFO:teuthology.orchestra.run.smithi121.stdout: "mdthresh_evicted": 0 2024-03-26T06:47:52.462 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.462 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2024-03-26T06:47:52.462 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 360, 2024-03-26T06:47:52.463 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 360, 2024-03-26T06:47:52.463 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:47:52.463 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:47:52.463 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:47:52.463 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:47:52.463 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:47:52.463 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:47:52.463 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:47:52.464 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:47:52.464 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:47:52.464 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:47:52.464 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:47:52.464 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:47:52.464 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:47:52.464 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2024-03-26T06:47:52.464 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:47:52.464 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2024-03-26T06:47:52.465 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:47:52.465 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:47:52.465 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:47:52.465 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:47:52.465 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:47:52.465 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:47:52.465 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:47:52.465 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2024-03-26T06:47:52.466 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:47:52.466 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:47:52.466 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:47:52.466 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2024-03-26T06:47:52.466 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2024-03-26T06:47:52.466 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2024-03-26T06:47:52.466 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:47:52.466 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:47:52.467 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:47:52.467 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:47:52.467 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 328122, 2024-03-26T06:47:52.467 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 477, 2024-03-26T06:47:52.467 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 1144, 2024-03-26T06:47:52.467 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 13, 2024-03-26T06:47:52.467 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2024-03-26T06:47:52.467 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2024-03-26T06:47:52.468 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:47:52.468 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2024-03-26T06:47:52.468 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:47:52.468 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2024-03-26T06:47:52.468 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 8352, 2024-03-26T06:47:52.468 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 190, 2024-03-26T06:47:52.468 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:47:52.468 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:47:52.468 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2024-03-26T06:47:52.469 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2024-03-26T06:47:52.469 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 2744807, 2024-03-26T06:47:52.469 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 22403, 2024-03-26T06:47:52.469 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2024-03-26T06:47:52.469 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2024-03-26T06:47:52.469 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2024-03-26T06:47:52.469 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2024-03-26T06:47:52.469 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.470 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter": { 2024-03-26T06:47:52.470 INFO:teuthology.orchestra.run.smithi121.stdout: "op_active": 0, 2024-03-26T06:47:52.470 INFO:teuthology.orchestra.run.smithi121.stdout: "op_laggy": 0, 2024-03-26T06:47:52.470 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send": 4440, 2024-03-26T06:47:52.470 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send_bytes": 68812178, 2024-03-26T06:47:52.470 INFO:teuthology.orchestra.run.smithi121.stdout: "op_resend": 37, 2024-03-26T06:47:52.470 INFO:teuthology.orchestra.run.smithi121.stdout: "op_reply": 4440, 2024-03-26T06:47:52.470 INFO:teuthology.orchestra.run.smithi121.stdout: "op_latency": { 2024-03-26T06:47:52.471 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 4440, 2024-03-26T06:47:52.471 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 304.846933541, 2024-03-26T06:47:52.471 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.068659219 2024-03-26T06:47:52.471 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.471 INFO:teuthology.orchestra.run.smithi121.stdout: "op_inflight": 0, 2024-03-26T06:47:52.471 INFO:teuthology.orchestra.run.smithi121.stdout: "oplen_avg": { 2024-03-26T06:47:52.471 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 4440, 2024-03-26T06:47:52.471 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 6222 2024-03-26T06:47:52.472 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.472 INFO:teuthology.orchestra.run.smithi121.stdout: "op": 4440, 2024-03-26T06:47:52.472 INFO:teuthology.orchestra.run.smithi121.stdout: "op_r": 6, 2024-03-26T06:47:52.472 INFO:teuthology.orchestra.run.smithi121.stdout: "op_w": 4434, 2024-03-26T06:47:52.472 INFO:teuthology.orchestra.run.smithi121.stdout: "op_rmw": 0, 2024-03-26T06:47:52.472 INFO:teuthology.orchestra.run.smithi121.stdout: "op_pg": 0, 2024-03-26T06:47:52.472 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_stat": 159, 2024-03-26T06:47:52.472 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_create": 609, 2024-03-26T06:47:52.473 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_read": 6, 2024-03-26T06:47:52.473 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_write": 2305, 2024-03-26T06:47:52.473 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writefull": 82, 2024-03-26T06:47:52.473 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writesame": 0, 2024-03-26T06:47:52.473 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_append": 0, 2024-03-26T06:47:52.473 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_zero": 0, 2024-03-26T06:47:52.473 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_truncate": 2, 2024-03-26T06:47:52.473 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_delete": 1011, 2024-03-26T06:47:52.474 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_mapext": 0, 2024-03-26T06:47:52.474 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_sparse_read": 0, 2024-03-26T06:47:52.474 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_clonerange": 0, 2024-03-26T06:47:52.474 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_getxattr": 0, 2024-03-26T06:47:52.474 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_setxattr": 1222, 2024-03-26T06:47:52.474 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_cmpxattr": 0, 2024-03-26T06:47:52.474 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_rmxattr": 0, 2024-03-26T06:47:52.474 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_resetxattrs": 0, 2024-03-26T06:47:52.475 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_call": 0, 2024-03-26T06:47:52.475 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_watch": 0, 2024-03-26T06:47:52.475 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_notify": 0, 2024-03-26T06:47:52.475 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_src_cmpxattr": 0, 2024-03-26T06:47:52.475 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls": 0, 2024-03-26T06:47:52.475 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls_filter": 0, 2024-03-26T06:47:52.475 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_other": 6, 2024-03-26T06:47:52.475 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_active": 0, 2024-03-26T06:47:52.476 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_send": 0, 2024-03-26T06:47:52.476 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_resend": 0, 2024-03-26T06:47:52.476 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_ping": 0, 2024-03-26T06:47:52.476 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_active": 0, 2024-03-26T06:47:52.476 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_send": 0, 2024-03-26T06:47:52.476 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_resend": 0, 2024-03-26T06:47:52.476 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_active": 0, 2024-03-26T06:47:52.476 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_send": 0, 2024-03-26T06:47:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_resend": 0, 2024-03-26T06:47:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_active": 0, 2024-03-26T06:47:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_send": 0, 2024-03-26T06:47:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_resend": 0, 2024-03-26T06:47:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: "command_active": 0, 2024-03-26T06:47:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: "command_send": 0, 2024-03-26T06:47:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: "command_resend": 0, 2024-03-26T06:47:52.477 INFO:teuthology.orchestra.run.smithi121.stdout: "map_epoch": 98, 2024-03-26T06:47:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: "map_full": 0, 2024-03-26T06:47:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: "map_inc": 13, 2024-03-26T06:47:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_sessions": 12, 2024-03-26T06:47:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_open": 12, 2024-03-26T06:47:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_close": 0, 2024-03-26T06:47:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_laggy": 0, 2024-03-26T06:47:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_wr": 703, 2024-03-26T06:47:52.478 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_rd": 0, 2024-03-26T06:47:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_del": 117 2024-03-26T06:47:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: "oft": { 2024-03-26T06:47:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_objs": 1, 2024-03-26T06:47:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_kv_pairs": 0, 2024-03-26T06:47:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_updates": 0, 2024-03-26T06:47:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_removes": 0 2024-03-26T06:47:52.479 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: "osd-slow-ops": { 2024-03-26T06:47:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_ops_count": 0 2024-03-26T06:47:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: "purge_queue": { 2024-03-26T06:47:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops": 0, 2024-03-26T06:47:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops_high_water": 10, 2024-03-26T06:47:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing": 0, 2024-03-26T06:47:52.480 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_high_water": 10, 2024-03-26T06:47:52.481 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executed_ops": 469, 2024-03-26T06:47:52.481 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executed": 469, 2024-03-26T06:47:52.481 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_item_in_journal": 0 2024-03-26T06:47:52.481 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.481 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:47:52.481 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:52.481 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2024-03-26T06:47:52.481 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:52.482 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 35301, 2024-03-26T06:47:52.482 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 9910684, 2024-03-26T06:47:52.482 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:52.482 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 35301, 2024-03-26T06:47:52.482 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-03-26T06:47:52.482 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-03-26T06:47:52.482 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 35301, 2024-03-26T06:47:52.482 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 9910684, 2024-03-26T06:47:52.483 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:52.483 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.483 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.483 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.483 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:52.483 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.483 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_bytes": { 2024-03-26T06:47:52.483 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:52.483 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2024-03-26T06:47:52.484 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:52.484 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2024-03-26T06:47:52.484 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2024-03-26T06:47:52.484 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:52.484 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2024-03-26T06:47:52.484 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 4440, 2024-03-26T06:47:52.484 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 68859715, 2024-03-26T06:47:52.484 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 4440, 2024-03-26T06:47:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 68859715, 2024-03-26T06:47:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.485 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_ops": { 2024-03-26T06:47:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 1024, 2024-03-26T06:47:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2024-03-26T06:47:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2024-03-26T06:47:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2024-03-26T06:47:52.486 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 4440, 2024-03-26T06:47:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 4440, 2024-03-26T06:47:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 4440, 2024-03-26T06:47:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 4440, 2024-03-26T06:47:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.487 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:52.488 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.488 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle": { 2024-03-26T06:47:52.488 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:52.488 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2024-03-26T06:47:52.488 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:52.488 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 469, 2024-03-26T06:47:52.488 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 47537, 2024-03-26T06:47:52.488 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:52.489 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 469, 2024-03-26T06:47:52.489 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-03-26T06:47:52.489 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-03-26T06:47:52.489 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 6, 2024-03-26T06:47:52.489 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 47537, 2024-03-26T06:47:52.490 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:52.490 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.490 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.490 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.490 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:52.490 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:52.490 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle-0x5612786146e0": { 2024-03-26T06:47:52.491 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:52.491 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2024-03-26T06:47:52.491 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:52.491 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 26930, 2024-03-26T06:47:52.491 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 66694140, 2024-03-26T06:47:52.491 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:52.492 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 26930, 2024-03-26T06:47:52.492 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-03-26T06:47:52.492 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-03-26T06:47:52.492 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 2299, 2024-03-26T06:47:52.493 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 66694140, 2024-03-26T06:47:52.493 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:52.493 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:52.494 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:52.494 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:52.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:52.494 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:52.494 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-03-26T06:47:52.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:52 smithi038 ceph-mon[32265]: pgmap v321: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-26T06:47:52.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:52 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:52.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:52 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:52.835 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:52 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.835 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:52 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.835 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:52 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.835 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:52 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.835 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:52 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:47:52.835 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:52 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:47:52.835 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:52 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:52.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:52 smithi121 ceph-mon[36349]: pgmap v321: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-26T06:47:52.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:52 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:52.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:52 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:52.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:52 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:52 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:52 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.884 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:52 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:52 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:47:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:52 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:47:52.885 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:52 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:52.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:52 smithi170 ceph-mon[36912]: pgmap v321: 97 pgs: 97 active+clean; 13 GiB data, 64 GiB used, 1009 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-26T06:47:52.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:52 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:52.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:52 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:52.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:52 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:52 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:52 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:52 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-26T06:47:52.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:52 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T06:47:52.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:52 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T06:47:52.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:52 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' 2024-03-26T06:47:53.340 INFO:tasks.check_counter:Daemon mds.b mds.exported=65 2024-03-26T06:47:53.340 INFO:tasks.check_counter:Daemon mds.b mds.imported=2 2024-03-26T06:47:53.341 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-26T06:47:53.341 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=10 2024-03-26T06:47:53.341 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=4 2024-03-26T06:47:53.341 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=3 2024-03-26T06:47:53.341 DEBUG:tasks.check_counter:Getting stats from e 2024-03-26T06:47:53.341 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-26T06:47:53.602 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:47:53.642 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:53 smithi121 ceph-mon[36349]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-03-26T06:47:53.643 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:53 smithi121 ceph-mon[36349]: Unable to set osd_memory_target on smithi038 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-26T06:47:53.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:53 smithi038 ceph-mon[32265]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-03-26T06:47:53.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:53 smithi038 ceph-mon[32265]: Unable to set osd_memory_target on smithi038 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-26T06:47:53.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:53 smithi170 ceph-mon[36912]: Adjusting osd_memory_target on smithi038 to 177.2M 2024-03-26T06:47:53.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:53 smithi170 ceph-mon[36912]: Unable to set osd_memory_target on smithi038 to 185895628: error parsing value: Value '185895628' is below minimum 939524096 2024-03-26T06:47:54.728 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:54 smithi121 ceph-mon[36349]: pgmap v322: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:47:54.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:54 smithi038 ceph-mon[32265]: pgmap v322: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:47:54.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:54 smithi170 ceph-mon[36912]: pgmap v322: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-26T06:47:55.325 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-03-26T06:47:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:47:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 2095, 2024-03-26T06:47:55.325 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 2097, 2024-03-26T06:47:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 624492, 2024-03-26T06:47:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 666217, 2024-03-26T06:47:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 6, 2024-03-26T06:47:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 2, 2024-03-26T06:47:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.187988088, 2024-03-26T06:47:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.046935285, 2024-03-26T06:47:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.043976769, 2024-03-26T06:47:55.326 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.085246978, 2024-03-26T06:47:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2095, 2024-03-26T06:47:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.064730292, 2024-03-26T06:47:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000030897 2024-03-26T06:47:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.327 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-26T06:47:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 0 2024-03-26T06:47:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:47:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 8, 2024-03-26T06:47:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 163, 2024-03-26T06:47:55.328 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 3900, 2024-03-26T06:47:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 439226, 2024-03-26T06:47:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 6, 2024-03-26T06:47:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 4, 2024-03-26T06:47:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.009670038, 2024-03-26T06:47:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.005919968, 2024-03-26T06:47:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.002055913, 2024-03-26T06:47:55.329 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000051429, 2024-03-26T06:47:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 161, 2024-03-26T06:47:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.017066434, 2024-03-26T06:47:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000106002 2024-03-26T06:47:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.330 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-26T06:47:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 438528 2024-03-26T06:47:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:47:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 245, 2024-03-26T06:47:55.331 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 226, 2024-03-26T06:47:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 1626876, 2024-03-26T06:47:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 128046, 2024-03-26T06:47:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 3, 2024-03-26T06:47:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 2, 2024-03-26T06:47:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.047441326, 2024-03-26T06:47:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.013122075, 2024-03-26T06:47:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.039715533, 2024-03-26T06:47:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.005155093, 2024-03-26T06:47:55.332 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 225, 2024-03-26T06:47:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.012524649, 2024-03-26T06:47:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000055665 2024-03-26T06:47:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.333 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1626608, 2024-03-26T06:47:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 127728 2024-03-26T06:47:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2024-03-26T06:47:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 1, 2024-03-26T06:47:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2024-03-26T06:47:55.334 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-MDSRank": { 2024-03-26T06:47:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2024-03-26T06:47:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2024-03-26T06:47:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 3565, 2024-03-26T06:47:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.083994404, 2024-03-26T06:47:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000023560 2024-03-26T06:47:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.335 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-PurgeQueue": { 2024-03-26T06:47:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2024-03-26T06:47:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2024-03-26T06:47:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-03-26T06:47:55.336 INFO:teuthology.orchestra.run.smithi121.stdout: "request": 0, 2024-03-26T06:47:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "reply": 0, 2024-03-26T06:47:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "reply_latency": { 2024-03-26T06:47:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_reply": 0, 2024-03-26T06:47:55.337 INFO:teuthology.orchestra.run.smithi121.stdout: "forward": 0, 2024-03-26T06:47:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_complete": 0, 2024-03-26T06:47:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_keys": 0, 2024-03-26T06:47:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_commit": 0, 2024-03-26T06:47:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_split": 0, 2024-03-26T06:47:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_merge": 0, 2024-03-26T06:47:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes": 15, 2024-03-26T06:47:55.338 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_top": 0, 2024-03-26T06:47:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_bottom": 15, 2024-03-26T06:47:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pin_tail": 0, 2024-03-26T06:47:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pinned": 15, 2024-03-26T06:47:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_expired": 0, 2024-03-26T06:47:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_with_caps": 0, 2024-03-26T06:47:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "caps": 0, 2024-03-26T06:47:55.339 INFO:teuthology.orchestra.run.smithi121.stdout: "subtrees": 4, 2024-03-26T06:47:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse": 0, 2024-03-26T06:47:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_hit": 0, 2024-03-26T06:47:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_forward": 0, 2024-03-26T06:47:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_discover": 0, 2024-03-26T06:47:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_dir_fetch": 0, 2024-03-26T06:47:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_remote_ino": 0, 2024-03-26T06:47:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_lock": 0, 2024-03-26T06:47:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "load_cent": 0, 2024-03-26T06:47:55.340 INFO:teuthology.orchestra.run.smithi121.stdout: "q": 0, 2024-03-26T06:47:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "exported": 0, 2024-03-26T06:47:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "exported_inodes": 0, 2024-03-26T06:47:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "imported": 0, 2024-03-26T06:47:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "imported_inodes": 0, 2024-03-26T06:47:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_dir_fetch": 0, 2024-03-26T06:47:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_backtrace_fetch": 0, 2024-03-26T06:47:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_peer_discover": 0, 2024-03-26T06:47:55.341 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rfiles": 0, 2024-03-26T06:47:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rbytes": 0, 2024-03-26T06:47:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rsnaps": 0, 2024-03-26T06:47:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_fetch": 0, 2024-03-26T06:47:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_set_tag": 0, 2024-03-26T06:47:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_repaired": 0, 2024-03-26T06:47:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_inotable_repaired": 0, 2024-03-26T06:47:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_inodes": 0, 2024-03-26T06:47:55.342 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_base_inodes": 0, 2024-03-26T06:47:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dirfrag_rstats": 0, 2024-03-26T06:47:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_file_inodes": 0, 2024-03-26T06:47:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_inode_file_caps": 0, 2024-03-26T06:47:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_revoke": 0, 2024-03-26T06:47:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_grant": 0, 2024-03-26T06:47:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_trunc": 0, 2024-03-26T06:47:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-26T06:47:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-26T06:47:55.343 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps": 0, 2024-03-26T06:47:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps_dirty": 0, 2024-03-26T06:47:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_cap_release": 0, 2024-03-26T06:47:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "process_request_cap_release": 0 2024-03-26T06:47:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache": { 2024-03-26T06:47:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update": 0, 2024-03-26T06:47:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update_receipt": 0, 2024-03-26T06:47:55.344 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_try_discover": 0, 2024-03-26T06:47:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_send_discover": 0, 2024-03-26T06:47:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_handle_discover": 0, 2024-03-26T06:47:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays": 0, 2024-03-26T06:47:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_delayed": 0, 2024-03-26T06:47:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_enqueuing": 0, 2024-03-26T06:47:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_created": 0, 2024-03-26T06:47:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_enqueued": 0, 2024-03-26T06:47:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_reintegrated": 0, 2024-03-26T06:47:55.345 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_migrated": 0, 2024-03-26T06:47:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_processing": 0, 2024-03-26T06:47:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_enqueued": 0, 2024-03-26T06:47:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_prioritized": 0, 2024-03-26T06:47:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_started": 0, 2024-03-26T06:47:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_completed": 0, 2024-03-26T06:47:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_enqueue_scrub": 0, 2024-03-26T06:47:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_exportdir": 0, 2024-03-26T06:47:55.346 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_flush": 0, 2024-03-26T06:47:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragmentdir": 0, 2024-03-26T06:47:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragstats": 0, 2024-03-26T06:47:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_inodestats": 0 2024-03-26T06:47:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log": { 2024-03-26T06:47:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "evlrg": 0, 2024-03-26T06:47:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "evadd": 0, 2024-03-26T06:47:55.347 INFO:teuthology.orchestra.run.smithi121.stdout: "evex": 0, 2024-03-26T06:47:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "evtrm": 0, 2024-03-26T06:47:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "ev": 14, 2024-03-26T06:47:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "evexg": 0, 2024-03-26T06:47:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "evexd": 0, 2024-03-26T06:47:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "segadd": 0, 2024-03-26T06:47:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "segex": 0, 2024-03-26T06:47:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "segtrm": 0, 2024-03-26T06:47:55.348 INFO:teuthology.orchestra.run.smithi121.stdout: "seg": 2, 2024-03-26T06:47:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "segmjr": 2, 2024-03-26T06:47:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "segexg": 0, 2024-03-26T06:47:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "segexd": 0, 2024-03-26T06:47:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "expos": 4194304, 2024-03-26T06:47:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "wrpos": 4224893, 2024-03-26T06:47:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "rdpos": 4224319, 2024-03-26T06:47:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "jlat": { 2024-03-26T06:47:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.349 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "replayed": 14 2024-03-26T06:47:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_mem": { 2024-03-26T06:47:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 16, 2024-03-26T06:47:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "ino+": 16, 2024-03-26T06:47:55.350 INFO:teuthology.orchestra.run.smithi121.stdout: "ino-": 0, 2024-03-26T06:47:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: "dir": 13, 2024-03-26T06:47:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: "dir+": 13, 2024-03-26T06:47:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: "dir-": 0, 2024-03-26T06:47:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: "dn": 15, 2024-03-26T06:47:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: "dn+": 15, 2024-03-26T06:47:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: "dn-": 0, 2024-03-26T06:47:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: "cap": 0, 2024-03-26T06:47:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: "cap+": 0, 2024-03-26T06:47:55.351 INFO:teuthology.orchestra.run.smithi121.stdout: "cap-": 0, 2024-03-26T06:47:55.352 INFO:teuthology.orchestra.run.smithi121.stdout: "rss": 47408, 2024-03-26T06:47:55.352 INFO:teuthology.orchestra.run.smithi121.stdout: "heap": 215420 2024-03-26T06:47:55.352 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.352 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_server": { 2024-03-26T06:47:55.352 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_client_request": 0, 2024-03-26T06:47:55.352 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_server_request": 0, 2024-03-26T06:47:55.352 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_request": 0, 2024-03-26T06:47:55.352 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_session": 0, 2024-03-26T06:47:55.353 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_peer_request": 0, 2024-03-26T06:47:55.353 INFO:teuthology.orchestra.run.smithi121.stdout: "req_create_latency": { 2024-03-26T06:47:55.353 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.353 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.353 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.353 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.353 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getattr_latency": { 2024-03-26T06:47:55.353 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.354 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.354 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.354 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getfilelock_latency": { 2024-03-26T06:47:55.354 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.354 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.354 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.354 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.355 INFO:teuthology.orchestra.run.smithi121.stdout: "req_link_latency": { 2024-03-26T06:47:55.355 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.355 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.355 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.355 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.355 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookup_latency": { 2024-03-26T06:47:55.355 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.355 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.355 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.356 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookuphash_latency": { 2024-03-26T06:47:55.356 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.356 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.356 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.356 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.356 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupino_latency": { 2024-03-26T06:47:55.356 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.356 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.357 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.357 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupname_latency": { 2024-03-26T06:47:55.357 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.357 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.357 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.357 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.358 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupparent_latency": { 2024-03-26T06:47:55.358 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.358 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.358 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.358 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.358 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupsnap_latency": { 2024-03-26T06:47:55.358 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.358 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.359 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.359 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lssnap_latency": { 2024-03-26T06:47:55.359 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.359 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.359 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.359 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.359 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mkdir_latency": { 2024-03-26T06:47:55.359 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.360 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.360 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.360 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.361 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mknod_latency": { 2024-03-26T06:47:55.361 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.361 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.361 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.361 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.361 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mksnap_latency": { 2024-03-26T06:47:55.361 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.362 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.362 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.362 INFO:teuthology.orchestra.run.smithi121.stdout: "req_open_latency": { 2024-03-26T06:47:55.362 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.362 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.362 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.362 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.363 INFO:teuthology.orchestra.run.smithi121.stdout: "req_readdir_latency": { 2024-03-26T06:47:55.363 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.363 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.363 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.363 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.363 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rename_latency": { 2024-03-26T06:47:55.363 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.363 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.364 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.364 INFO:teuthology.orchestra.run.smithi121.stdout: "req_renamesnap_latency": { 2024-03-26T06:47:55.364 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.364 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.364 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.364 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.364 INFO:teuthology.orchestra.run.smithi121.stdout: "req_snapdiff_latency": { 2024-03-26T06:47:55.364 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.365 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.365 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.365 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmdir_latency": { 2024-03-26T06:47:55.365 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.365 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.365 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.365 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.366 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmsnap_latency": { 2024-03-26T06:47:55.366 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.366 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.366 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.366 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.366 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmxattr_latency": { 2024-03-26T06:47:55.366 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.366 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.367 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.367 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setattr_latency": { 2024-03-26T06:47:55.367 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.367 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.367 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.367 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.367 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setdirlayout_latency": { 2024-03-26T06:47:55.368 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.368 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.368 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.368 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.368 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setfilelock_latency": { 2024-03-26T06:47:55.368 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.368 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.368 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.369 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setlayout_latency": { 2024-03-26T06:47:55.369 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.369 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.369 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.369 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.369 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setxattr_latency": { 2024-03-26T06:47:55.369 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.370 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.370 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.370 INFO:teuthology.orchestra.run.smithi121.stdout: "req_symlink_latency": { 2024-03-26T06:47:55.370 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.370 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.370 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.370 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.370 INFO:teuthology.orchestra.run.smithi121.stdout: "req_unlink_latency": { 2024-03-26T06:47:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_revoke_eviction": 0, 2024-03-26T06:47:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_acquisition_throttle": 0, 2024-03-26T06:47:55.371 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getvxattr_latency": { 2024-03-26T06:47:55.372 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.372 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.372 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.372 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:55.372 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.372 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_sessions": { 2024-03-26T06:47:55.372 INFO:teuthology.orchestra.run.smithi121.stdout: "session_count": 2, 2024-03-26T06:47:55.372 INFO:teuthology.orchestra.run.smithi121.stdout: "session_add": 3, 2024-03-26T06:47:55.373 INFO:teuthology.orchestra.run.smithi121.stdout: "session_remove": 1, 2024-03-26T06:47:55.373 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_open": 3, 2024-03-26T06:47:55.373 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_stale": 0, 2024-03-26T06:47:55.373 INFO:teuthology.orchestra.run.smithi121.stdout: "total_load": 0, 2024-03-26T06:47:55.373 INFO:teuthology.orchestra.run.smithi121.stdout: "average_load": 0, 2024-03-26T06:47:55.373 INFO:teuthology.orchestra.run.smithi121.stdout: "avg_session_uptime": 2900, 2024-03-26T06:47:55.373 INFO:teuthology.orchestra.run.smithi121.stdout: "mdthresh_evicted": 0 2024-03-26T06:47:55.373 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.373 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2024-03-26T06:47:55.374 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 0, 2024-03-26T06:47:55.374 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 0, 2024-03-26T06:47:55.374 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:47:55.374 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:47:55.374 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:47:55.374 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:47:55.374 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:47:55.374 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:47:55.375 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:47:55.375 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:47:55.375 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:47:55.375 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:47:55.375 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:47:55.375 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:47:55.375 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:47:55.375 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2024-03-26T06:47:55.376 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:47:55.376 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2024-03-26T06:47:55.376 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:47:55.376 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:47:55.376 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:47:55.376 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:47:55.376 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:47:55.376 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:47:55.377 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:47:55.377 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2024-03-26T06:47:55.377 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:47:55.377 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:47:55.377 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:47:55.377 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2024-03-26T06:47:55.377 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2024-03-26T06:47:55.377 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2024-03-26T06:47:55.377 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:47:55.378 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:47:55.378 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:47:55.378 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:47:55.378 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 181661, 2024-03-26T06:47:55.378 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 57, 2024-03-26T06:47:55.378 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 0, 2024-03-26T06:47:55.378 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 0, 2024-03-26T06:47:55.378 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2024-03-26T06:47:55.379 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2024-03-26T06:47:55.379 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:47:55.379 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2024-03-26T06:47:55.379 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:47:55.379 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2024-03-26T06:47:55.379 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 8352, 2024-03-26T06:47:55.379 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 190, 2024-03-26T06:47:55.379 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:47:55.379 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:47:55.380 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2024-03-26T06:47:55.380 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2024-03-26T06:47:55.380 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 67411, 2024-03-26T06:47:55.380 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 673, 2024-03-26T06:47:55.380 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2024-03-26T06:47:55.380 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2024-03-26T06:47:55.380 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2024-03-26T06:47:55.380 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2024-03-26T06:47:55.381 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.381 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter": { 2024-03-26T06:47:55.381 INFO:teuthology.orchestra.run.smithi121.stdout: "op_active": 0, 2024-03-26T06:47:55.381 INFO:teuthology.orchestra.run.smithi121.stdout: "op_laggy": 0, 2024-03-26T06:47:55.381 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send": 2098, 2024-03-26T06:47:55.381 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send_bytes": 21, 2024-03-26T06:47:55.381 INFO:teuthology.orchestra.run.smithi121.stdout: "op_resend": 3, 2024-03-26T06:47:55.381 INFO:teuthology.orchestra.run.smithi121.stdout: "op_reply": 2098, 2024-03-26T06:47:55.381 INFO:teuthology.orchestra.run.smithi121.stdout: "op_latency": { 2024-03-26T06:47:55.382 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2098, 2024-03-26T06:47:55.382 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 14.242810337, 2024-03-26T06:47:55.382 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.006788756 2024-03-26T06:47:55.382 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.382 INFO:teuthology.orchestra.run.smithi121.stdout: "op_inflight": 0, 2024-03-26T06:47:55.382 INFO:teuthology.orchestra.run.smithi121.stdout: "oplen_avg": { 2024-03-26T06:47:55.382 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2098, 2024-03-26T06:47:55.382 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 2099 2024-03-26T06:47:55.383 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.383 INFO:teuthology.orchestra.run.smithi121.stdout: "op": 2098, 2024-03-26T06:47:55.383 INFO:teuthology.orchestra.run.smithi121.stdout: "op_r": 2098, 2024-03-26T06:47:55.383 INFO:teuthology.orchestra.run.smithi121.stdout: "op_w": 0, 2024-03-26T06:47:55.383 INFO:teuthology.orchestra.run.smithi121.stdout: "op_rmw": 0, 2024-03-26T06:47:55.383 INFO:teuthology.orchestra.run.smithi121.stdout: "op_pg": 0, 2024-03-26T06:47:55.383 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_stat": 1349, 2024-03-26T06:47:55.383 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_create": 0, 2024-03-26T06:47:55.384 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_read": 747, 2024-03-26T06:47:55.384 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_write": 0, 2024-03-26T06:47:55.384 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writefull": 0, 2024-03-26T06:47:55.384 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writesame": 0, 2024-03-26T06:47:55.384 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_append": 0, 2024-03-26T06:47:55.384 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_zero": 0, 2024-03-26T06:47:55.384 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_truncate": 0, 2024-03-26T06:47:55.384 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_delete": 0, 2024-03-26T06:47:55.384 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_mapext": 0, 2024-03-26T06:47:55.385 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_sparse_read": 0, 2024-03-26T06:47:55.385 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_clonerange": 0, 2024-03-26T06:47:55.385 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_getxattr": 1, 2024-03-26T06:47:55.385 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_setxattr": 0, 2024-03-26T06:47:55.385 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_cmpxattr": 0, 2024-03-26T06:47:55.385 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_rmxattr": 0, 2024-03-26T06:47:55.385 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_resetxattrs": 0, 2024-03-26T06:47:55.385 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_call": 0, 2024-03-26T06:47:55.386 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_watch": 0, 2024-03-26T06:47:55.386 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_notify": 0, 2024-03-26T06:47:55.386 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_src_cmpxattr": 0, 2024-03-26T06:47:55.386 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls": 0, 2024-03-26T06:47:55.386 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls_filter": 0, 2024-03-26T06:47:55.386 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_other": 0, 2024-03-26T06:47:55.386 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_active": 0, 2024-03-26T06:47:55.386 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_send": 0, 2024-03-26T06:47:55.386 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_resend": 0, 2024-03-26T06:47:55.387 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_ping": 0, 2024-03-26T06:47:55.387 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_active": 0, 2024-03-26T06:47:55.387 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_send": 0, 2024-03-26T06:47:55.387 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_resend": 0, 2024-03-26T06:47:55.387 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_active": 0, 2024-03-26T06:47:55.387 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_send": 0, 2024-03-26T06:47:55.387 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_resend": 0, 2024-03-26T06:47:55.387 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_active": 0, 2024-03-26T06:47:55.388 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_send": 0, 2024-03-26T06:47:55.388 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_resend": 0, 2024-03-26T06:47:55.388 INFO:teuthology.orchestra.run.smithi121.stdout: "command_active": 0, 2024-03-26T06:47:55.388 INFO:teuthology.orchestra.run.smithi121.stdout: "command_send": 0, 2024-03-26T06:47:55.388 INFO:teuthology.orchestra.run.smithi121.stdout: "command_resend": 0, 2024-03-26T06:47:55.388 INFO:teuthology.orchestra.run.smithi121.stdout: "map_epoch": 98, 2024-03-26T06:47:55.388 INFO:teuthology.orchestra.run.smithi121.stdout: "map_full": 0, 2024-03-26T06:47:55.388 INFO:teuthology.orchestra.run.smithi121.stdout: "map_inc": 12, 2024-03-26T06:47:55.388 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_sessions": 4, 2024-03-26T06:47:55.389 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_open": 4, 2024-03-26T06:47:55.389 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_close": 0, 2024-03-26T06:47:55.389 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_laggy": 0, 2024-03-26T06:47:55.389 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_wr": 0, 2024-03-26T06:47:55.389 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_rd": 2, 2024-03-26T06:47:55.389 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_del": 0 2024-03-26T06:47:55.389 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.389 INFO:teuthology.orchestra.run.smithi121.stdout: "oft": { 2024-03-26T06:47:55.390 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_objs": 0, 2024-03-26T06:47:55.390 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_kv_pairs": 0, 2024-03-26T06:47:55.390 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_updates": 0, 2024-03-26T06:47:55.390 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_removes": 0 2024-03-26T06:47:55.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.390 INFO:teuthology.orchestra.run.smithi121.stdout: "osd-slow-ops": { 2024-03-26T06:47:55.390 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_ops_count": 0 2024-03-26T06:47:55.390 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.391 INFO:teuthology.orchestra.run.smithi121.stdout: "purge_queue": { 2024-03-26T06:47:55.391 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops": 0, 2024-03-26T06:47:55.391 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops_high_water": 0, 2024-03-26T06:47:55.391 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing": 0, 2024-03-26T06:47:55.391 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_high_water": 0, 2024-03-26T06:47:55.391 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executed_ops": 0, 2024-03-26T06:47:55.391 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executed": 0, 2024-03-26T06:47:55.391 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_item_in_journal": 0 2024-03-26T06:47:55.391 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.392 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:47:55.392 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:55.392 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2024-03-26T06:47:55.392 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:55.392 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 2344, 2024-03-26T06:47:55.392 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 2028873, 2024-03-26T06:47:55.392 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:55.392 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 2344, 2024-03-26T06:47:55.393 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-03-26T06:47:55.393 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-03-26T06:47:55.393 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 2344, 2024-03-26T06:47:55.393 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 2028873, 2024-03-26T06:47:55.393 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:55.393 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.393 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.393 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.393 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:55.394 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.394 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_bytes": { 2024-03-26T06:47:55.394 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:55.394 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2024-03-26T06:47:55.394 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:55.394 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2024-03-26T06:47:55.394 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2024-03-26T06:47:55.394 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:55.395 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2024-03-26T06:47:55.395 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 2098, 2024-03-26T06:47:55.395 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 30594, 2024-03-26T06:47:55.395 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 2098, 2024-03-26T06:47:55.395 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 30594, 2024-03-26T06:47:55.395 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:55.395 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.395 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.395 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.396 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:55.396 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.396 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_ops": { 2024-03-26T06:47:55.396 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:55.396 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 1024, 2024-03-26T06:47:55.396 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:55.396 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2024-03-26T06:47:55.396 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2024-03-26T06:47:55.397 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:55.397 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2024-03-26T06:47:55.397 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 2098, 2024-03-26T06:47:55.397 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 2098, 2024-03-26T06:47:55.397 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 2098, 2024-03-26T06:47:55.397 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 2098, 2024-03-26T06:47:55.397 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:55.397 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.397 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.398 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.398 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:55.398 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.398 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle": { 2024-03-26T06:47:55.398 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:55.398 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2024-03-26T06:47:55.398 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:55.399 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2024-03-26T06:47:55.399 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2024-03-26T06:47:55.399 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:55.399 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2024-03-26T06:47:55.399 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-03-26T06:47:55.399 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-03-26T06:47:55.399 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 0, 2024-03-26T06:47:55.399 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 0, 2024-03-26T06:47:55.399 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:55.400 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.400 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.400 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.400 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:55.400 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:55.400 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle-0x5593fd40e960": { 2024-03-26T06:47:55.400 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:55.400 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2024-03-26T06:47:55.400 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:55.401 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2024-03-26T06:47:55.401 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2024-03-26T06:47:55.401 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:55.401 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2024-03-26T06:47:55.401 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-03-26T06:47:55.401 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-03-26T06:47:55.401 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 0, 2024-03-26T06:47:55.401 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 0, 2024-03-26T06:47:55.402 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:55.402 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:55.402 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:55.402 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:55.402 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:55.402 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:55.402 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-03-26T06:47:56.056 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-26T06:47:56.056 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-26T06:47:56.056 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=0 2024-03-26T06:47:56.056 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-03-26T06:47:56.056 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-03-26T06:47:56.056 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-26T06:47:56.057 DEBUG:tasks.check_counter:Getting stats from h 2024-03-26T06:47:56.057 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-26T06:47:56.298 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:47:56.670 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:56 smithi121 ceph-mon[36349]: pgmap v323: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:47:56.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:56 smithi038 ceph-mon[32265]: pgmap v323: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:47:56.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:56 smithi170 ceph-mon[36912]: pgmap v323: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:47:57.909 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-03-26T06:47:57.909 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:47:57.909 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 148, 2024-03-26T06:47:57.910 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 284, 2024-03-26T06:47:57.910 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 48670, 2024-03-26T06:47:57.910 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 1581001, 2024-03-26T06:47:57.910 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 14, 2024-03-26T06:47:57.910 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 5, 2024-03-26T06:47:57.910 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.023021069, 2024-03-26T06:47:57.910 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.009860856, 2024-03-26T06:47:57.911 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.006743453, 2024-03-26T06:47:57.911 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.003553528, 2024-03-26T06:47:57.911 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:57.911 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 280, 2024-03-26T06:47:57.911 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.038878516, 2024-03-26T06:47:57.911 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000138851 2024-03-26T06:47:57.911 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.911 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:57.911 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 54, 2024-03-26T06:47:57.912 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000008560, 2024-03-26T06:47:57.912 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000158 2024-03-26T06:47:57.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.912 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 2704, 2024-03-26T06:47:57.912 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 311088 2024-03-26T06:47:57.912 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.912 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:47:57.913 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 244, 2024-03-26T06:47:57.913 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 518, 2024-03-26T06:47:57.913 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 71486, 2024-03-26T06:47:57.913 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 422744, 2024-03-26T06:47:57.913 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 11, 2024-03-26T06:47:57.913 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 5, 2024-03-26T06:47:57.913 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.024505702, 2024-03-26T06:47:57.913 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.009940824, 2024-03-26T06:47:57.914 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.014717763, 2024-03-26T06:47:57.914 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000619828, 2024-03-26T06:47:57.914 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:57.914 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 516, 2024-03-26T06:47:57.914 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.046413655, 2024-03-26T06:47:57.914 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000089948 2024-03-26T06:47:57.914 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.914 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:57.915 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 442, 2024-03-26T06:47:57.915 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000102449, 2024-03-26T06:47:57.915 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000231 2024-03-26T06:47:57.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.915 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 5040, 2024-03-26T06:47:57.915 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 130848 2024-03-26T06:47:57.915 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.915 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:47:57.915 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 1669, 2024-03-26T06:47:57.916 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 1446, 2024-03-26T06:47:57.916 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 2303677, 2024-03-26T06:47:57.916 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 597701, 2024-03-26T06:47:57.916 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 6, 2024-03-26T06:47:57.916 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 4, 2024-03-26T06:47:57.916 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.138992329, 2024-03-26T06:47:57.916 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.056741495, 2024-03-26T06:47:57.916 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.098156587, 2024-03-26T06:47:57.917 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.009486943, 2024-03-26T06:47:57.917 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:47:57.917 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1447, 2024-03-26T06:47:57.917 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.072787706, 2024-03-26T06:47:57.917 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000050302 2024-03-26T06:47:57.917 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.918 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:47:57.918 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2443, 2024-03-26T06:47:57.918 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000454227, 2024-03-26T06:47:57.918 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000185 2024-03-26T06:47:57.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.918 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1637600, 2024-03-26T06:47:57.918 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 137392 2024-03-26T06:47:57.918 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.919 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2024-03-26T06:47:57.919 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 1, 2024-03-26T06:47:57.919 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2024-03-26T06:47:57.919 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.919 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-MDSRank": { 2024-03-26T06:47:57.919 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2024-03-26T06:47:57.919 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2024-03-26T06:47:57.919 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 227, 2024-03-26T06:47:57.920 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.060651475, 2024-03-26T06:47:57.920 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000267187 2024-03-26T06:47:57.920 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:57.920 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.920 INFO:teuthology.orchestra.run.smithi121.stdout: "finisher-PurgeQueue": { 2024-03-26T06:47:57.920 INFO:teuthology.orchestra.run.smithi121.stdout: "queue_len": 0, 2024-03-26T06:47:57.920 INFO:teuthology.orchestra.run.smithi121.stdout: "complete_latency": { 2024-03-26T06:47:57.920 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2, 2024-03-26T06:47:57.921 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000011222, 2024-03-26T06:47:57.921 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000005611 2024-03-26T06:47:57.921 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:57.921 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.921 INFO:teuthology.orchestra.run.smithi121.stdout: "mds": { 2024-03-26T06:47:57.921 INFO:teuthology.orchestra.run.smithi121.stdout: "request": 93, 2024-03-26T06:47:57.921 INFO:teuthology.orchestra.run.smithi121.stdout: "reply": 75, 2024-03-26T06:47:57.921 INFO:teuthology.orchestra.run.smithi121.stdout: "reply_latency": { 2024-03-26T06:47:57.922 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 75, 2024-03-26T06:47:57.922 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.034650367, 2024-03-26T06:47:57.922 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000462004 2024-03-26T06:47:57.922 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.922 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_reply": 0, 2024-03-26T06:47:57.922 INFO:teuthology.orchestra.run.smithi121.stdout: "forward": 18, 2024-03-26T06:47:57.922 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_complete": 0, 2024-03-26T06:47:57.922 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_fetch_keys": 0, 2024-03-26T06:47:57.923 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_commit": 11, 2024-03-26T06:47:57.923 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_split": 0, 2024-03-26T06:47:57.923 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_merge": 0, 2024-03-26T06:47:57.923 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes": 260, 2024-03-26T06:47:57.923 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_top": 2, 2024-03-26T06:47:57.923 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_bottom": 22, 2024-03-26T06:47:57.923 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pin_tail": 236, 2024-03-26T06:47:57.923 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_pinned": 258, 2024-03-26T06:47:57.924 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_expired": 3, 2024-03-26T06:47:57.924 INFO:teuthology.orchestra.run.smithi121.stdout: "inodes_with_caps": 156, 2024-03-26T06:47:57.924 INFO:teuthology.orchestra.run.smithi121.stdout: "caps": 156, 2024-03-26T06:47:57.924 INFO:teuthology.orchestra.run.smithi121.stdout: "subtrees": 8, 2024-03-26T06:47:57.924 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse": 163, 2024-03-26T06:47:57.924 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_hit": 85, 2024-03-26T06:47:57.924 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_forward": 11, 2024-03-26T06:47:57.924 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_discover": 63, 2024-03-26T06:47:57.924 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_dir_fetch": 0, 2024-03-26T06:47:57.925 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_remote_ino": 0, 2024-03-26T06:47:57.925 INFO:teuthology.orchestra.run.smithi121.stdout: "traverse_lock": 0, 2024-03-26T06:47:57.925 INFO:teuthology.orchestra.run.smithi121.stdout: "load_cent": 0, 2024-03-26T06:47:57.925 INFO:teuthology.orchestra.run.smithi121.stdout: "q": 0, 2024-03-26T06:47:57.925 INFO:teuthology.orchestra.run.smithi121.stdout: "exported": 2, 2024-03-26T06:47:57.925 INFO:teuthology.orchestra.run.smithi121.stdout: "exported_inodes": 2, 2024-03-26T06:47:57.925 INFO:teuthology.orchestra.run.smithi121.stdout: "imported": 63, 2024-03-26T06:47:57.926 INFO:teuthology.orchestra.run.smithi121.stdout: "imported_inodes": 223, 2024-03-26T06:47:57.926 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_dir_fetch": 0, 2024-03-26T06:47:57.926 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_backtrace_fetch": 0, 2024-03-26T06:47:57.926 INFO:teuthology.orchestra.run.smithi121.stdout: "openino_peer_discover": 0, 2024-03-26T06:47:57.926 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rfiles": 324, 2024-03-26T06:47:57.926 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rbytes": 13424234229, 2024-03-26T06:47:57.926 INFO:teuthology.orchestra.run.smithi121.stdout: "root_rsnaps": 5, 2024-03-26T06:47:57.926 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_fetch": 0, 2024-03-26T06:47:57.926 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_set_tag": 0, 2024-03-26T06:47:57.927 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_backtrace_repaired": 0, 2024-03-26T06:47:57.927 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_inotable_repaired": 0, 2024-03-26T06:47:57.927 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_inodes": 0, 2024-03-26T06:47:57.927 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dir_base_inodes": 0, 2024-03-26T06:47:57.927 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_dirfrag_rstats": 0, 2024-03-26T06:47:57.927 INFO:teuthology.orchestra.run.smithi121.stdout: "scrub_file_inodes": 0, 2024-03-26T06:47:57.927 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_inode_file_caps": 5, 2024-03-26T06:47:57.928 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_revoke": 5, 2024-03-26T06:47:57.928 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_grant": 125, 2024-03-26T06:47:57.928 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_trunc": 0, 2024-03-26T06:47:57.928 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-26T06:47:57.928 INFO:teuthology.orchestra.run.smithi121.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-26T06:47:57.928 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps": 112, 2024-03-26T06:47:57.928 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_caps_dirty": 42, 2024-03-26T06:47:57.928 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_cap_release": 0, 2024-03-26T06:47:57.928 INFO:teuthology.orchestra.run.smithi121.stdout: "process_request_cap_release": 72 2024-03-26T06:47:57.929 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.929 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_cache": { 2024-03-26T06:47:57.929 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update": 0, 2024-03-26T06:47:57.929 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_update_receipt": 0, 2024-03-26T06:47:57.929 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_try_discover": 0, 2024-03-26T06:47:57.929 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_send_discover": 70, 2024-03-26T06:47:57.929 INFO:teuthology.orchestra.run.smithi121.stdout: "dir_handle_discover": 0, 2024-03-26T06:47:57.929 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays": 0, 2024-03-26T06:47:57.930 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_delayed": 0, 2024-03-26T06:47:57.930 INFO:teuthology.orchestra.run.smithi121.stdout: "num_strays_enqueuing": 0, 2024-03-26T06:47:57.930 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_created": 0, 2024-03-26T06:47:57.930 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_enqueued": 0, 2024-03-26T06:47:57.930 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_reintegrated": 0, 2024-03-26T06:47:57.930 INFO:teuthology.orchestra.run.smithi121.stdout: "strays_migrated": 0, 2024-03-26T06:47:57.930 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_processing": 0, 2024-03-26T06:47:57.930 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_enqueued": 0, 2024-03-26T06:47:57.931 INFO:teuthology.orchestra.run.smithi121.stdout: "num_recovering_prioritized": 0, 2024-03-26T06:47:57.931 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_started": 0, 2024-03-26T06:47:57.931 INFO:teuthology.orchestra.run.smithi121.stdout: "recovery_completed": 0, 2024-03-26T06:47:57.931 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_enqueue_scrub": 0, 2024-03-26T06:47:57.931 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_exportdir": 2, 2024-03-26T06:47:57.931 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_flush": 0, 2024-03-26T06:47:57.931 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragmentdir": 0, 2024-03-26T06:47:57.931 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_fragstats": 0, 2024-03-26T06:47:57.932 INFO:teuthology.orchestra.run.smithi121.stdout: "ireq_inodestats": 0 2024-03-26T06:47:57.932 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.932 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_log": { 2024-03-26T06:47:57.932 INFO:teuthology.orchestra.run.smithi121.stdout: "evlrg": 0, 2024-03-26T06:47:57.932 INFO:teuthology.orchestra.run.smithi121.stdout: "evadd": 264, 2024-03-26T06:47:57.932 INFO:teuthology.orchestra.run.smithi121.stdout: "evex": 0, 2024-03-26T06:47:57.932 INFO:teuthology.orchestra.run.smithi121.stdout: "evtrm": 0, 2024-03-26T06:47:57.932 INFO:teuthology.orchestra.run.smithi121.stdout: "ev": 264, 2024-03-26T06:47:57.933 INFO:teuthology.orchestra.run.smithi121.stdout: "evexg": 0, 2024-03-26T06:47:57.933 INFO:teuthology.orchestra.run.smithi121.stdout: "evexd": 0, 2024-03-26T06:47:57.933 INFO:teuthology.orchestra.run.smithi121.stdout: "segadd": 2, 2024-03-26T06:47:57.933 INFO:teuthology.orchestra.run.smithi121.stdout: "segex": 0, 2024-03-26T06:47:57.933 INFO:teuthology.orchestra.run.smithi121.stdout: "segtrm": 0, 2024-03-26T06:47:57.933 INFO:teuthology.orchestra.run.smithi121.stdout: "seg": 2, 2024-03-26T06:47:57.933 INFO:teuthology.orchestra.run.smithi121.stdout: "segmjr": 2, 2024-03-26T06:47:57.933 INFO:teuthology.orchestra.run.smithi121.stdout: "segexg": 0, 2024-03-26T06:47:57.933 INFO:teuthology.orchestra.run.smithi121.stdout: "segexd": 0, 2024-03-26T06:47:57.934 INFO:teuthology.orchestra.run.smithi121.stdout: "expos": 4194304, 2024-03-26T06:47:57.934 INFO:teuthology.orchestra.run.smithi121.stdout: "wrpos": 5390237, 2024-03-26T06:47:57.934 INFO:teuthology.orchestra.run.smithi121.stdout: "rdpos": 0, 2024-03-26T06:47:57.934 INFO:teuthology.orchestra.run.smithi121.stdout: "jlat": { 2024-03-26T06:47:57.934 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 99, 2024-03-26T06:47:57.934 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.809743144, 2024-03-26T06:47:57.934 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.008179223 2024-03-26T06:47:57.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.935 INFO:teuthology.orchestra.run.smithi121.stdout: "replayed": 0 2024-03-26T06:47:57.935 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.935 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_mem": { 2024-03-26T06:47:57.935 INFO:teuthology.orchestra.run.smithi121.stdout: "ino": 196, 2024-03-26T06:47:57.935 INFO:teuthology.orchestra.run.smithi121.stdout: "ino+": 196, 2024-03-26T06:47:57.935 INFO:teuthology.orchestra.run.smithi121.stdout: "ino-": 0, 2024-03-26T06:47:57.935 INFO:teuthology.orchestra.run.smithi121.stdout: "dir": 100, 2024-03-26T06:47:57.935 INFO:teuthology.orchestra.run.smithi121.stdout: "dir+": 102, 2024-03-26T06:47:57.936 INFO:teuthology.orchestra.run.smithi121.stdout: "dir-": 2, 2024-03-26T06:47:57.936 INFO:teuthology.orchestra.run.smithi121.stdout: "dn": 260, 2024-03-26T06:47:57.936 INFO:teuthology.orchestra.run.smithi121.stdout: "dn+": 263, 2024-03-26T06:47:57.936 INFO:teuthology.orchestra.run.smithi121.stdout: "dn-": 3, 2024-03-26T06:47:57.936 INFO:teuthology.orchestra.run.smithi121.stdout: "cap": 156, 2024-03-26T06:47:57.936 INFO:teuthology.orchestra.run.smithi121.stdout: "cap+": 159, 2024-03-26T06:47:57.936 INFO:teuthology.orchestra.run.smithi121.stdout: "cap-": 3, 2024-03-26T06:47:57.936 INFO:teuthology.orchestra.run.smithi121.stdout: "rss": 53708, 2024-03-26T06:47:57.937 INFO:teuthology.orchestra.run.smithi121.stdout: "heap": 207228 2024-03-26T06:47:57.937 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.937 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_server": { 2024-03-26T06:47:57.937 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_client_request": 99, 2024-03-26T06:47:57.937 INFO:teuthology.orchestra.run.smithi121.stdout: "dispatch_server_request": 1, 2024-03-26T06:47:57.937 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_request": 93, 2024-03-26T06:47:57.937 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_client_session": 27, 2024-03-26T06:47:57.937 INFO:teuthology.orchestra.run.smithi121.stdout: "handle_peer_request": 2, 2024-03-26T06:47:57.938 INFO:teuthology.orchestra.run.smithi121.stdout: "req_create_latency": { 2024-03-26T06:47:57.938 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.938 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.938 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.938 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.938 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getattr_latency": { 2024-03-26T06:47:57.938 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 6, 2024-03-26T06:47:57.938 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.003449793, 2024-03-26T06:47:57.938 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000574965 2024-03-26T06:47:57.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.939 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getfilelock_latency": { 2024-03-26T06:47:57.939 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.939 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.939 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.939 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.939 INFO:teuthology.orchestra.run.smithi121.stdout: "req_link_latency": { 2024-03-26T06:47:57.939 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.940 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.940 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.940 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.940 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookup_latency": { 2024-03-26T06:47:57.940 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 2, 2024-03-26T06:47:57.940 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000765034, 2024-03-26T06:47:57.940 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000382517 2024-03-26T06:47:57.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.941 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookuphash_latency": { 2024-03-26T06:47:57.941 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.941 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.941 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.941 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.941 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupino_latency": { 2024-03-26T06:47:57.941 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 1, 2024-03-26T06:47:57.942 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000110934, 2024-03-26T06:47:57.942 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000110934 2024-03-26T06:47:57.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.942 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupname_latency": { 2024-03-26T06:47:57.942 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.942 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.942 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.942 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.943 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupparent_latency": { 2024-03-26T06:47:57.943 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.943 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.943 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.943 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.943 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lookupsnap_latency": { 2024-03-26T06:47:57.943 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.943 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.943 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.944 INFO:teuthology.orchestra.run.smithi121.stdout: "req_lssnap_latency": { 2024-03-26T06:47:57.944 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.944 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.944 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.944 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.944 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mkdir_latency": { 2024-03-26T06:47:57.944 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.945 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.945 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.945 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.946 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mknod_latency": { 2024-03-26T06:47:57.946 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.946 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.946 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.946 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.946 INFO:teuthology.orchestra.run.smithi121.stdout: "req_mksnap_latency": { 2024-03-26T06:47:57.946 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.946 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.947 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.947 INFO:teuthology.orchestra.run.smithi121.stdout: "req_open_latency": { 2024-03-26T06:47:57.947 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.947 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.947 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.947 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.947 INFO:teuthology.orchestra.run.smithi121.stdout: "req_readdir_latency": { 2024-03-26T06:47:57.948 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 66, 2024-03-26T06:47:57.948 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.030324606, 2024-03-26T06:47:57.948 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000459463 2024-03-26T06:47:57.948 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.948 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rename_latency": { 2024-03-26T06:47:57.948 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.948 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.948 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.949 INFO:teuthology.orchestra.run.smithi121.stdout: "req_renamesnap_latency": { 2024-03-26T06:47:57.949 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.949 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.949 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.949 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.949 INFO:teuthology.orchestra.run.smithi121.stdout: "req_snapdiff_latency": { 2024-03-26T06:47:57.949 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.949 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.950 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.950 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmdir_latency": { 2024-03-26T06:47:57.950 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.950 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.950 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.950 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.950 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmsnap_latency": { 2024-03-26T06:47:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: "req_rmxattr_latency": { 2024-03-26T06:47:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.951 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setattr_latency": { 2024-03-26T06:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setdirlayout_latency": { 2024-03-26T06:47:57.952 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setfilelock_latency": { 2024-03-26T06:47:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.953 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setlayout_latency": { 2024-03-26T06:47:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "req_setxattr_latency": { 2024-03-26T06:47:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.954 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "req_symlink_latency": { 2024-03-26T06:47:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "req_unlink_latency": { 2024-03-26T06:47:57.955 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_revoke_eviction": 0, 2024-03-26T06:47:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "cap_acquisition_throttle": 0, 2024-03-26T06:47:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "req_getvxattr_latency": { 2024-03-26T06:47:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.956 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_sessions": { 2024-03-26T06:47:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "session_count": 2, 2024-03-26T06:47:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "session_add": 5, 2024-03-26T06:47:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "session_remove": 3, 2024-03-26T06:47:57.957 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_open": 2, 2024-03-26T06:47:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "sessions_stale": 0, 2024-03-26T06:47:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "total_load": 71, 2024-03-26T06:47:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "average_load": 23, 2024-03-26T06:47:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "avg_session_uptime": 607, 2024-03-26T06:47:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "mdthresh_evicted": 0 2024-03-26T06:47:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2024-03-26T06:47:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 0, 2024-03-26T06:47:57.958 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 0, 2024-03-26T06:47:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:47:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:47:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:47:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:47:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:47:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:47:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:47:57.959 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:47:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:47:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:47:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:47:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:47:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:47:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2024-03-26T06:47:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:47:57.960 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2024-03-26T06:47:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:47:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:47:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:47:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:47:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:47:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:47:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:47:57.961 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2024-03-26T06:47:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:47:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:47:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:47:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2024-03-26T06:47:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2024-03-26T06:47:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2024-03-26T06:47:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:47:57.962 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:47:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:47:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:47:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 272668, 2024-03-26T06:47:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 261, 2024-03-26T06:47:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 0, 2024-03-26T06:47:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 0, 2024-03-26T06:47:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2024-03-26T06:47:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2024-03-26T06:47:57.963 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:47:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2024-03-26T06:47:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:47:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2024-03-26T06:47:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 8352, 2024-03-26T06:47:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 190, 2024-03-26T06:47:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:47:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:47:57.964 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2024-03-26T06:47:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2024-03-26T06:47:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 913424, 2024-03-26T06:47:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 18097, 2024-03-26T06:47:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2024-03-26T06:47:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2024-03-26T06:47:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2024-03-26T06:47:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2024-03-26T06:47:57.965 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "objecter": { 2024-03-26T06:47:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "op_active": 0, 2024-03-26T06:47:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "op_laggy": 0, 2024-03-26T06:47:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send": 147, 2024-03-26T06:47:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "op_send_bytes": 1220188, 2024-03-26T06:47:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "op_resend": 33, 2024-03-26T06:47:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "op_reply": 147, 2024-03-26T06:47:57.966 INFO:teuthology.orchestra.run.smithi121.stdout: "op_latency": { 2024-03-26T06:47:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 147, 2024-03-26T06:47:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 1.341982025, 2024-03-26T06:47:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.009129129 2024-03-26T06:47:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "op_inflight": 0, 2024-03-26T06:47:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "oplen_avg": { 2024-03-26T06:47:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 147, 2024-03-26T06:47:57.967 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 177 2024-03-26T06:47:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "op": 147, 2024-03-26T06:47:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "op_r": 0, 2024-03-26T06:47:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "op_w": 147, 2024-03-26T06:47:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "op_rmw": 0, 2024-03-26T06:47:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "op_pg": 0, 2024-03-26T06:47:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_stat": 0, 2024-03-26T06:47:57.968 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_create": 10, 2024-03-26T06:47:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_read": 0, 2024-03-26T06:47:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_write": 99, 2024-03-26T06:47:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writefull": 11, 2024-03-26T06:47:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_writesame": 0, 2024-03-26T06:47:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_append": 0, 2024-03-26T06:47:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_zero": 0, 2024-03-26T06:47:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_truncate": 0, 2024-03-26T06:47:57.969 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_delete": 6, 2024-03-26T06:47:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_mapext": 0, 2024-03-26T06:47:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_sparse_read": 0, 2024-03-26T06:47:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_clonerange": 0, 2024-03-26T06:47:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_getxattr": 0, 2024-03-26T06:47:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_setxattr": 20, 2024-03-26T06:47:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_cmpxattr": 0, 2024-03-26T06:47:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_rmxattr": 0, 2024-03-26T06:47:57.970 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_resetxattrs": 0, 2024-03-26T06:47:57.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_call": 0, 2024-03-26T06:47:57.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_watch": 0, 2024-03-26T06:47:57.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_notify": 0, 2024-03-26T06:47:57.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_src_cmpxattr": 0, 2024-03-26T06:47:57.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls": 0, 2024-03-26T06:47:57.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_pgls_filter": 0, 2024-03-26T06:47:57.971 INFO:teuthology.orchestra.run.smithi121.stdout: "osdop_other": 0, 2024-03-26T06:47:57.971 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_active": 0, 2024-03-26T06:47:57.971 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_send": 0, 2024-03-26T06:47:57.972 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_resend": 0, 2024-03-26T06:47:57.972 INFO:teuthology.orchestra.run.smithi121.stdout: "linger_ping": 0, 2024-03-26T06:47:57.972 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_active": 0, 2024-03-26T06:47:57.972 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_send": 0, 2024-03-26T06:47:57.972 INFO:teuthology.orchestra.run.smithi121.stdout: "poolop_resend": 0, 2024-03-26T06:47:57.972 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_active": 0, 2024-03-26T06:47:57.972 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_send": 0, 2024-03-26T06:47:57.972 INFO:teuthology.orchestra.run.smithi121.stdout: "poolstat_resend": 0, 2024-03-26T06:47:57.973 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_active": 0, 2024-03-26T06:47:57.973 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_send": 0, 2024-03-26T06:47:57.973 INFO:teuthology.orchestra.run.smithi121.stdout: "statfs_resend": 0, 2024-03-26T06:47:57.973 INFO:teuthology.orchestra.run.smithi121.stdout: "command_active": 0, 2024-03-26T06:47:57.973 INFO:teuthology.orchestra.run.smithi121.stdout: "command_send": 0, 2024-03-26T06:47:57.973 INFO:teuthology.orchestra.run.smithi121.stdout: "command_resend": 0, 2024-03-26T06:47:57.973 INFO:teuthology.orchestra.run.smithi121.stdout: "map_epoch": 98, 2024-03-26T06:47:57.973 INFO:teuthology.orchestra.run.smithi121.stdout: "map_full": 0, 2024-03-26T06:47:57.973 INFO:teuthology.orchestra.run.smithi121.stdout: "map_inc": 12, 2024-03-26T06:47:57.974 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_sessions": 8, 2024-03-26T06:47:57.974 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_open": 8, 2024-03-26T06:47:57.974 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_session_close": 0, 2024-03-26T06:47:57.974 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_laggy": 0, 2024-03-26T06:47:57.974 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_wr": 28, 2024-03-26T06:47:57.974 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_rd": 0, 2024-03-26T06:47:57.974 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_del": 3 2024-03-26T06:47:57.974 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.975 INFO:teuthology.orchestra.run.smithi121.stdout: "oft": { 2024-03-26T06:47:57.975 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_objs": 1, 2024-03-26T06:47:57.975 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_kv_pairs": 0, 2024-03-26T06:47:57.975 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_updates": 0, 2024-03-26T06:47:57.975 INFO:teuthology.orchestra.run.smithi121.stdout: "omap_total_removes": 0 2024-03-26T06:47:57.975 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.975 INFO:teuthology.orchestra.run.smithi121.stdout: "osd-slow-ops": { 2024-03-26T06:47:57.975 INFO:teuthology.orchestra.run.smithi121.stdout: "slow_ops_count": 0 2024-03-26T06:47:57.976 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.976 INFO:teuthology.orchestra.run.smithi121.stdout: "purge_queue": { 2024-03-26T06:47:57.976 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops": 0, 2024-03-26T06:47:57.976 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_ops_high_water": 0, 2024-03-26T06:47:57.976 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing": 0, 2024-03-26T06:47:57.976 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executing_high_water": 0, 2024-03-26T06:47:57.976 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executed_ops": 0, 2024-03-26T06:47:57.976 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_executed": 0, 2024-03-26T06:47:57.977 INFO:teuthology.orchestra.run.smithi121.stdout: "pq_item_in_journal": 0 2024-03-26T06:47:57.977 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.977 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:47:57.977 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:57.977 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2024-03-26T06:47:57.977 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:57.977 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 2057, 2024-03-26T06:47:57.977 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 2224130, 2024-03-26T06:47:57.977 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:57.978 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 2057, 2024-03-26T06:47:57.978 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-03-26T06:47:57.978 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-03-26T06:47:57.978 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 2057, 2024-03-26T06:47:57.978 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 2224130, 2024-03-26T06:47:57.978 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:57.978 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.978 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.979 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.979 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:57.979 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.979 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_bytes": { 2024-03-26T06:47:57.979 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:57.979 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2024-03-26T06:47:57.979 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:57.979 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2024-03-26T06:47:57.980 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2024-03-26T06:47:57.980 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:57.980 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2024-03-26T06:47:57.980 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 147, 2024-03-26T06:47:57.980 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 1220188, 2024-03-26T06:47:57.980 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 147, 2024-03-26T06:47:57.980 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 1220188, 2024-03-26T06:47:57.980 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:57.981 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.981 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.981 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.981 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:57.981 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.981 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-objecter_ops": { 2024-03-26T06:47:57.981 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:57.981 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 1024, 2024-03-26T06:47:57.982 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:57.982 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2024-03-26T06:47:57.982 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2024-03-26T06:47:57.982 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:57.982 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2024-03-26T06:47:57.982 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 147, 2024-03-26T06:47:57.982 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 147, 2024-03-26T06:47:57.982 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 147, 2024-03-26T06:47:57.982 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 147, 2024-03-26T06:47:57.983 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:57.983 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.983 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.983 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.983 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:57.983 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.983 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle": { 2024-03-26T06:47:57.983 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:57.984 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2024-03-26T06:47:57.984 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:57.984 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 0, 2024-03-26T06:47:57.984 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 0, 2024-03-26T06:47:57.984 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:57.984 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 0, 2024-03-26T06:47:57.984 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-03-26T06:47:57.984 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-03-26T06:47:57.984 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 0, 2024-03-26T06:47:57.985 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 0, 2024-03-26T06:47:57.985 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:57.985 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.985 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.985 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.985 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:57.985 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:47:57.985 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-write_buf_throttle-0x558ba2a2e6e0": { 2024-03-26T06:47:57.986 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:47:57.986 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 3758096384, 2024-03-26T06:47:57.986 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:47:57.986 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 264, 2024-03-26T06:47:57.986 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 1195933, 2024-03-26T06:47:57.986 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:47:57.986 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 264, 2024-03-26T06:47:57.986 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-03-26T06:47:57.987 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-03-26T06:47:57.987 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 99, 2024-03-26T06:47:57.987 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 1195933, 2024-03-26T06:47:57.987 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:47:57.987 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:47:57.987 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:47:57.987 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:47:57.987 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:57.988 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:47:57.988 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-03-26T06:47:58.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:47:58 smithi121 ceph-mon[36349]: pgmap v324: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:47:58.738 INFO:tasks.check_counter:Daemon mds.h mds.exported=2 2024-03-26T06:47:58.738 INFO:tasks.check_counter:Daemon mds.h mds.imported=63 2024-03-26T06:47:58.739 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=5 2024-03-26T06:47:58.739 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-26T06:47:58.739 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-26T06:47:58.739 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-26T06:47:58.739 DEBUG:tasks.check_counter:Getting stats from k 2024-03-26T06:47:58.739 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-26T06:47:58.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:47:58 smithi038 ceph-mon[32265]: pgmap v324: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:47:58.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:47:58 smithi170 ceph-mon[36912]: pgmap v324: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-26T06:47:58.979 INFO:teuthology.orchestra.run.smithi121.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T06:48:00.616 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:00 smithi121 ceph-mon[36349]: pgmap v325: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:48:00.617 INFO:teuthology.orchestra.run.smithi121.stdout:{ 2024-03-26T06:48:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:48:00.617 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 4, 2024-03-26T06:48:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 4, 2024-03-26T06:48:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 3024, 2024-03-26T06:48:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 448, 2024-03-26T06:48:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 4, 2024-03-26T06:48:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 0, 2024-03-26T06:48:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.001436945, 2024-03-26T06:48:00.618 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.000030045, 2024-03-26T06:48:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.000624904, 2024-03-26T06:48:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:48:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 3, 2024-03-26T06:48:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000687110, 2024-03-26T06:48:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000229036 2024-03-26T06:48:00.619 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:48:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:48:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:48:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:48:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:48:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-26T06:48:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 448 2024-03-26T06:48:00.620 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:48:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:48:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 2, 2024-03-26T06:48:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 162, 2024-03-26T06:48:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 320, 2024-03-26T06:48:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 183600, 2024-03-26T06:48:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 5, 2024-03-26T06:48:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 3, 2024-03-26T06:48:00.621 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.010393138, 2024-03-26T06:48:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.006715078, 2024-03-26T06:48:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.001516457, 2024-03-26T06:48:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:48:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 159, 2024-03-26T06:48:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.019281262, 2024-03-26T06:48:00.622 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000121265 2024-03-26T06:48:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:48:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:48:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:48:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:48:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:48:00.623 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-26T06:48:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 183600 2024-03-26T06:48:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:48:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:48:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_messages": 222, 2024-03-26T06:48:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages": 211, 2024-03-26T06:48:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_bytes": 1574848, 2024-03-26T06:48:00.624 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_bytes": 126656, 2024-03-26T06:48:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_created_connections": 2, 2024-03-26T06:48:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_active_connections": 1, 2024-03-26T06:48:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_total_time": 0.046419200, 2024-03-26T06:48:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_send_time": 0.012842130, 2024-03-26T06:48:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_recv_time": 0.037840790, 2024-03-26T06:48:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_running_fast_dispatch_time": 0.005296474, 2024-03-26T06:48:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:00.625 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 210, 2024-03-26T06:48:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.012344308, 2024-03-26T06:48:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000058782 2024-03-26T06:48:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:48:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:48:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:48:00.626 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:48:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:48:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_recv_encrypted_bytes": 1574848, 2024-03-26T06:48:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "msgr_send_encrypted_bytes": 126656 2024-03-26T06:48:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:48:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "cct": { 2024-03-26T06:48:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "total_workers": 0, 2024-03-26T06:48:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: "unhealthy_workers": 0 2024-03-26T06:48:00.627 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:48:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "mempool": { 2024-03-26T06:48:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_bytes": 0, 2024-03-26T06:48:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "bloom_filter_items": 0, 2024-03-26T06:48:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:48:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:48:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:48:00.628 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:48:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:48:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:48:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:48:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:48:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:48:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:48:00.629 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:48:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:48:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:48:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_extent_items": 0, 2024-03-26T06:48:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:48:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_blob_items": 0, 2024-03-26T06:48:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:48:00.630 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:48:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:48:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:48:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:48:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:48:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:48:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_txc_items": 0, 2024-03-26T06:48:00.631 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:48:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:48:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:48:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "bluestore_writing_items": 0, 2024-03-26T06:48:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_bytes": 0, 2024-03-26T06:48:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_items": 0, 2024-03-26T06:48:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:48:00.632 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:48:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:48:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:48:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_bytes": 149047, 2024-03-26T06:48:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_anon_items": 31, 2024-03-26T06:48:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_bytes": 0, 2024-03-26T06:48:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "buffer_meta_items": 0, 2024-03-26T06:48:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_bytes": 0, 2024-03-26T06:48:00.633 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_items": 0, 2024-03-26T06:48:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:48:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_mapbl_items": 0, 2024-03-26T06:48:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:48:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "osd_pglog_items": 0, 2024-03-26T06:48:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_bytes": 0, 2024-03-26T06:48:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_items": 0, 2024-03-26T06:48:00.634 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:48:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:48:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_bytes": 0, 2024-03-26T06:48:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "pgmap_items": 0, 2024-03-26T06:48:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_bytes": 928, 2024-03-26T06:48:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "mds_co_items": 2, 2024-03-26T06:48:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_bytes": 0, 2024-03-26T06:48:00.635 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_1_items": 0, 2024-03-26T06:48:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_bytes": 0, 2024-03-26T06:48:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "unittest_2_items": 0 2024-03-26T06:48:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: }, 2024-03-26T06:48:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:48:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "val": 0, 2024-03-26T06:48:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "max": 104857600, 2024-03-26T06:48:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "get_started": 0, 2024-03-26T06:48:00.636 INFO:teuthology.orchestra.run.smithi121.stdout: "get": 224, 2024-03-26T06:48:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "get_sum": 1543578, 2024-03-26T06:48:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "get_or_fail_success": 224, 2024-03-26T06:48:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "take": 0, 2024-03-26T06:48:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "take_sum": 0, 2024-03-26T06:48:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "put": 224, 2024-03-26T06:48:00.637 INFO:teuthology.orchestra.run.smithi121.stdout: "put_sum": 1543578, 2024-03-26T06:48:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: "wait": { 2024-03-26T06:48:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: "avgcount": 0, 2024-03-26T06:48:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: "sum": 0.000000000, 2024-03-26T06:48:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: "avgtime": 0.000000000 2024-03-26T06:48:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:48:00.638 INFO:teuthology.orchestra.run.smithi121.stdout: } 2024-03-26T06:48:00.638 INFO:teuthology.orchestra.run.smithi121.stdout:} 2024-03-26T06:48:00.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:00 smithi038 ceph-mon[32265]: pgmap v325: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:48:00.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:00 smithi170 ceph-mon[36912]: pgmap v325: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-26T06:48:01.201 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-26T06:48:01.201 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-26T06:48:01.201 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-26T06:48:01.201 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-26T06:48:01.201 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-26T06:48:01.201 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-03-26T06:48:01.201 DEBUG:tasks.check_counter:Getting stats from c 2024-03-26T06:48:01.202 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-26T06:48:01.452 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:48:02.795 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:02 smithi170 ceph-mon[36912]: pgmap v326: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:48:02.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:02 smithi038 ceph-mon[32265]: pgmap v326: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:48:02.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:02 smithi121 ceph-mon[36349]: pgmap v326: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-26T06:48:03.156 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-26T06:48:03.156 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:48:03.157 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 1105, 2024-03-26T06:48:03.157 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 1107, 2024-03-26T06:48:03.157 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 29590414, 2024-03-26T06:48:03.157 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 351400, 2024-03-26T06:48:03.157 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 7, 2024-03-26T06:48:03.157 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 3, 2024-03-26T06:48:03.157 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.137015451, 2024-03-26T06:48:03.157 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.033675340, 2024-03-26T06:48:03.158 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.050493055, 2024-03-26T06:48:03.158 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.041958278, 2024-03-26T06:48:03.158 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:03.158 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1105, 2024-03-26T06:48:03.158 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.059805583, 2024-03-26T06:48:03.158 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000054122 2024-03-26T06:48:03.158 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.158 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:03.159 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.159 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.159 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.159 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.159 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-26T06:48:03.159 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 0 2024-03-26T06:48:03.159 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.159 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:48:03.160 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 369, 2024-03-26T06:48:03.160 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 362, 2024-03-26T06:48:03.160 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 21391802, 2024-03-26T06:48:03.160 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 157834, 2024-03-26T06:48:03.160 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 5, 2024-03-26T06:48:03.160 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 4, 2024-03-26T06:48:03.160 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.058139937, 2024-03-26T06:48:03.160 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.014691608, 2024-03-26T06:48:03.161 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.032025617, 2024-03-26T06:48:03.161 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.011302307, 2024-03-26T06:48:03.161 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:03.161 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 361, 2024-03-26T06:48:03.161 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.038048332, 2024-03-26T06:48:03.161 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000105397 2024-03-26T06:48:03.161 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.161 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:03.161 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.162 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.162 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.162 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.162 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 362752, 2024-03-26T06:48:03.162 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 97984 2024-03-26T06:48:03.166 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.166 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:48:03.166 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 549, 2024-03-26T06:48:03.166 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 667, 2024-03-26T06:48:03.166 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 16511975, 2024-03-26T06:48:03.166 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 470097, 2024-03-26T06:48:03.166 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 5, 2024-03-26T06:48:03.166 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 4, 2024-03-26T06:48:03.167 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.068750081, 2024-03-26T06:48:03.167 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.025235791, 2024-03-26T06:48:03.167 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.022048020, 2024-03-26T06:48:03.167 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.015326327, 2024-03-26T06:48:03.167 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:03.167 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 666, 2024-03-26T06:48:03.167 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.042898231, 2024-03-26T06:48:03.167 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000064411 2024-03-26T06:48:03.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.168 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:03.168 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.168 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.168 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.168 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 3184, 2024-03-26T06:48:03.168 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 297104 2024-03-26T06:48:03.168 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.169 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-03-26T06:48:03.169 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 1, 2024-03-26T06:48:03.169 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-03-26T06:48:03.169 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.169 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-MDSRank": { 2024-03-26T06:48:03.169 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-26T06:48:03.169 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-26T06:48:03.169 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 3021, 2024-03-26T06:48:03.169 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.112358568, 2024-03-26T06:48:03.170 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000037192 2024-03-26T06:48:03.170 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:03.170 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.170 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-PurgeQueue": { 2024-03-26T06:48:03.170 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-26T06:48:03.170 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-26T06:48:03.170 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.170 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.171 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.171 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:03.171 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.171 INFO:teuthology.orchestra.run.smithi170.stdout: "mds": { 2024-03-26T06:48:03.171 INFO:teuthology.orchestra.run.smithi170.stdout: "request": 0, 2024-03-26T06:48:03.171 INFO:teuthology.orchestra.run.smithi170.stdout: "reply": 0, 2024-03-26T06:48:03.171 INFO:teuthology.orchestra.run.smithi170.stdout: "reply_latency": { 2024-03-26T06:48:03.171 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.172 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.172 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.172 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.172 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_reply": 0, 2024-03-26T06:48:03.172 INFO:teuthology.orchestra.run.smithi170.stdout: "forward": 0, 2024-03-26T06:48:03.172 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_complete": 0, 2024-03-26T06:48:03.172 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_keys": 0, 2024-03-26T06:48:03.172 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_commit": 0, 2024-03-26T06:48:03.173 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_split": 0, 2024-03-26T06:48:03.173 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_merge": 0, 2024-03-26T06:48:03.173 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes": 1360, 2024-03-26T06:48:03.173 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_top": 665, 2024-03-26T06:48:03.173 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_bottom": 695, 2024-03-26T06:48:03.173 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pin_tail": 0, 2024-03-26T06:48:03.173 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pinned": 410, 2024-03-26T06:48:03.173 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_expired": 0, 2024-03-26T06:48:03.173 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_with_caps": 0, 2024-03-26T06:48:03.174 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": 0, 2024-03-26T06:48:03.174 INFO:teuthology.orchestra.run.smithi170.stdout: "subtrees": 6, 2024-03-26T06:48:03.174 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse": 0, 2024-03-26T06:48:03.174 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_hit": 0, 2024-03-26T06:48:03.174 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_forward": 0, 2024-03-26T06:48:03.174 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_discover": 0, 2024-03-26T06:48:03.174 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_dir_fetch": 0, 2024-03-26T06:48:03.174 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_remote_ino": 0, 2024-03-26T06:48:03.175 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_lock": 0, 2024-03-26T06:48:03.175 INFO:teuthology.orchestra.run.smithi170.stdout: "load_cent": 0, 2024-03-26T06:48:03.175 INFO:teuthology.orchestra.run.smithi170.stdout: "q": 0, 2024-03-26T06:48:03.175 INFO:teuthology.orchestra.run.smithi170.stdout: "exported": 0, 2024-03-26T06:48:03.175 INFO:teuthology.orchestra.run.smithi170.stdout: "exported_inodes": 0, 2024-03-26T06:48:03.175 INFO:teuthology.orchestra.run.smithi170.stdout: "imported": 0, 2024-03-26T06:48:03.175 INFO:teuthology.orchestra.run.smithi170.stdout: "imported_inodes": 0, 2024-03-26T06:48:03.175 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_dir_fetch": 0, 2024-03-26T06:48:03.176 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_backtrace_fetch": 0, 2024-03-26T06:48:03.176 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_peer_discover": 0, 2024-03-26T06:48:03.176 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rfiles": 324, 2024-03-26T06:48:03.176 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rbytes": 13424234229, 2024-03-26T06:48:03.176 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rsnaps": 6, 2024-03-26T06:48:03.176 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_fetch": 0, 2024-03-26T06:48:03.176 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_set_tag": 0, 2024-03-26T06:48:03.176 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_repaired": 0, 2024-03-26T06:48:03.176 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_inotable_repaired": 0, 2024-03-26T06:48:03.177 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_inodes": 0, 2024-03-26T06:48:03.177 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_base_inodes": 0, 2024-03-26T06:48:03.177 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dirfrag_rstats": 0, 2024-03-26T06:48:03.177 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_file_inodes": 0, 2024-03-26T06:48:03.177 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_inode_file_caps": 0, 2024-03-26T06:48:03.177 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_revoke": 0, 2024-03-26T06:48:03.177 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_grant": 0, 2024-03-26T06:48:03.177 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_trunc": 0, 2024-03-26T06:48:03.177 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-26T06:48:03.178 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-26T06:48:03.178 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps": 0, 2024-03-26T06:48:03.178 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps_dirty": 0, 2024-03-26T06:48:03.178 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_cap_release": 0, 2024-03-26T06:48:03.178 INFO:teuthology.orchestra.run.smithi170.stdout: "process_request_cap_release": 0 2024-03-26T06:48:03.178 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.178 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache": { 2024-03-26T06:48:03.178 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update": 0, 2024-03-26T06:48:03.179 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update_receipt": 0, 2024-03-26T06:48:03.179 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_try_discover": 0, 2024-03-26T06:48:03.179 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_send_discover": 0, 2024-03-26T06:48:03.179 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_handle_discover": 0, 2024-03-26T06:48:03.179 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays": 0, 2024-03-26T06:48:03.179 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_delayed": 0, 2024-03-26T06:48:03.179 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_enqueuing": 0, 2024-03-26T06:48:03.179 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_created": 0, 2024-03-26T06:48:03.179 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_enqueued": 0, 2024-03-26T06:48:03.180 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_reintegrated": 0, 2024-03-26T06:48:03.180 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_migrated": 0, 2024-03-26T06:48:03.180 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_processing": 0, 2024-03-26T06:48:03.180 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_enqueued": 0, 2024-03-26T06:48:03.180 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_prioritized": 0, 2024-03-26T06:48:03.180 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_started": 0, 2024-03-26T06:48:03.180 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_completed": 0, 2024-03-26T06:48:03.180 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_enqueue_scrub": 0, 2024-03-26T06:48:03.181 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_exportdir": 0, 2024-03-26T06:48:03.181 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_flush": 0, 2024-03-26T06:48:03.181 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragmentdir": 0, 2024-03-26T06:48:03.181 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragstats": 0, 2024-03-26T06:48:03.181 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_inodestats": 0 2024-03-26T06:48:03.181 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.181 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log": { 2024-03-26T06:48:03.181 INFO:teuthology.orchestra.run.smithi170.stdout: "evlrg": 0, 2024-03-26T06:48:03.182 INFO:teuthology.orchestra.run.smithi170.stdout: "evadd": 0, 2024-03-26T06:48:03.182 INFO:teuthology.orchestra.run.smithi170.stdout: "evex": 0, 2024-03-26T06:48:03.182 INFO:teuthology.orchestra.run.smithi170.stdout: "evtrm": 0, 2024-03-26T06:48:03.182 INFO:teuthology.orchestra.run.smithi170.stdout: "ev": 26887, 2024-03-26T06:48:03.182 INFO:teuthology.orchestra.run.smithi170.stdout: "evexg": 0, 2024-03-26T06:48:03.182 INFO:teuthology.orchestra.run.smithi170.stdout: "evexd": 0, 2024-03-26T06:48:03.182 INFO:teuthology.orchestra.run.smithi170.stdout: "segadd": 0, 2024-03-26T06:48:03.182 INFO:teuthology.orchestra.run.smithi170.stdout: "segex": 0, 2024-03-26T06:48:03.182 INFO:teuthology.orchestra.run.smithi170.stdout: "segtrm": 0, 2024-03-26T06:48:03.183 INFO:teuthology.orchestra.run.smithi170.stdout: "seg": 2, 2024-03-26T06:48:03.183 INFO:teuthology.orchestra.run.smithi170.stdout: "segmjr": 16, 2024-03-26T06:48:03.183 INFO:teuthology.orchestra.run.smithi170.stdout: "segexg": 0, 2024-03-26T06:48:03.183 INFO:teuthology.orchestra.run.smithi170.stdout: "segexd": 0, 2024-03-26T06:48:03.183 INFO:teuthology.orchestra.run.smithi170.stdout: "expos": 65025290, 2024-03-26T06:48:03.183 INFO:teuthology.orchestra.run.smithi170.stdout: "wrpos": 70893542, 2024-03-26T06:48:03.183 INFO:teuthology.orchestra.run.smithi170.stdout: "rdpos": 70893458, 2024-03-26T06:48:03.183 INFO:teuthology.orchestra.run.smithi170.stdout: "jlat": { 2024-03-26T06:48:03.183 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.184 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.184 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.184 INFO:teuthology.orchestra.run.smithi170.stdout: "replayed": 26887 2024-03-26T06:48:03.184 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.184 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mem": { 2024-03-26T06:48:03.184 INFO:teuthology.orchestra.run.smithi170.stdout: "ino": 765, 2024-03-26T06:48:03.184 INFO:teuthology.orchestra.run.smithi170.stdout: "ino+": 1386, 2024-03-26T06:48:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: "ino-": 621, 2024-03-26T06:48:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: "dir": 46, 2024-03-26T06:48:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: "dir+": 134, 2024-03-26T06:48:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: "dir-": 88, 2024-03-26T06:48:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: "dn": 1360, 2024-03-26T06:48:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: "dn+": 1629, 2024-03-26T06:48:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: "dn-": 269, 2024-03-26T06:48:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: "cap": 0, 2024-03-26T06:48:03.185 INFO:teuthology.orchestra.run.smithi170.stdout: "cap+": 0, 2024-03-26T06:48:03.186 INFO:teuthology.orchestra.run.smithi170.stdout: "cap-": 0, 2024-03-26T06:48:03.186 INFO:teuthology.orchestra.run.smithi170.stdout: "rss": 54008, 2024-03-26T06:48:03.186 INFO:teuthology.orchestra.run.smithi170.stdout: "heap": 215420 2024-03-26T06:48:03.186 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.186 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_server": { 2024-03-26T06:48:03.186 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_client_request": 0, 2024-03-26T06:48:03.186 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_server_request": 0, 2024-03-26T06:48:03.186 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_request": 0, 2024-03-26T06:48:03.186 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_session": 0, 2024-03-26T06:48:03.187 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_peer_request": 0, 2024-03-26T06:48:03.187 INFO:teuthology.orchestra.run.smithi170.stdout: "req_create_latency": { 2024-03-26T06:48:03.187 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.187 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.187 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.187 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.187 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getattr_latency": { 2024-03-26T06:48:03.187 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.188 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.188 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.188 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getfilelock_latency": { 2024-03-26T06:48:03.188 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.188 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.188 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.188 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.189 INFO:teuthology.orchestra.run.smithi170.stdout: "req_link_latency": { 2024-03-26T06:48:03.189 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.189 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.189 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.189 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.189 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookup_latency": { 2024-03-26T06:48:03.189 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.189 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.189 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.190 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookuphash_latency": { 2024-03-26T06:48:03.190 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.190 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.190 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.190 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.190 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupino_latency": { 2024-03-26T06:48:03.190 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.191 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.191 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.191 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupname_latency": { 2024-03-26T06:48:03.191 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.191 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.191 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.191 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.192 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupparent_latency": { 2024-03-26T06:48:03.192 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.192 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.192 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.192 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.192 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupsnap_latency": { 2024-03-26T06:48:03.192 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.192 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.193 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.193 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lssnap_latency": { 2024-03-26T06:48:03.193 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.193 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.193 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.193 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.193 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mkdir_latency": { 2024-03-26T06:48:03.193 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.194 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.194 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.194 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.195 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mknod_latency": { 2024-03-26T06:48:03.195 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.195 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.195 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.195 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.195 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mksnap_latency": { 2024-03-26T06:48:03.195 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.195 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.196 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.196 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.196 INFO:teuthology.orchestra.run.smithi170.stdout: "req_open_latency": { 2024-03-26T06:48:03.196 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.196 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.196 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.196 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.196 INFO:teuthology.orchestra.run.smithi170.stdout: "req_readdir_latency": { 2024-03-26T06:48:03.196 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.197 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.197 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.197 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.197 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rename_latency": { 2024-03-26T06:48:03.197 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.197 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.197 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.197 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.198 INFO:teuthology.orchestra.run.smithi170.stdout: "req_renamesnap_latency": { 2024-03-26T06:48:03.198 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.198 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.198 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.198 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.199 INFO:teuthology.orchestra.run.smithi170.stdout: "req_snapdiff_latency": { 2024-03-26T06:48:03.199 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.199 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.199 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.199 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.199 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmdir_latency": { 2024-03-26T06:48:03.199 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.199 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.199 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.199 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.200 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmsnap_latency": { 2024-03-26T06:48:03.200 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.200 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.200 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.200 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.200 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmxattr_latency": { 2024-03-26T06:48:03.200 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.200 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.201 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.201 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.201 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setattr_latency": { 2024-03-26T06:48:03.201 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.201 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.201 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.201 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.201 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setdirlayout_latency": { 2024-03-26T06:48:03.202 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.202 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.202 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.202 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.202 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setfilelock_latency": { 2024-03-26T06:48:03.202 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.202 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.202 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.203 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setlayout_latency": { 2024-03-26T06:48:03.203 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.203 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.203 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.203 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.203 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setxattr_latency": { 2024-03-26T06:48:03.203 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.204 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.204 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.204 INFO:teuthology.orchestra.run.smithi170.stdout: "req_symlink_latency": { 2024-03-26T06:48:03.204 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.204 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.204 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.204 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.204 INFO:teuthology.orchestra.run.smithi170.stdout: "req_unlink_latency": { 2024-03-26T06:48:03.205 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.205 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.205 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.205 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.205 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_revoke_eviction": 0, 2024-03-26T06:48:03.205 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_acquisition_throttle": 0, 2024-03-26T06:48:03.205 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getvxattr_latency": { 2024-03-26T06:48:03.205 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.206 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.206 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.206 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:03.206 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.206 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessions": { 2024-03-26T06:48:03.206 INFO:teuthology.orchestra.run.smithi170.stdout: "session_count": 2, 2024-03-26T06:48:03.206 INFO:teuthology.orchestra.run.smithi170.stdout: "session_add": 3, 2024-03-26T06:48:03.207 INFO:teuthology.orchestra.run.smithi170.stdout: "session_remove": 1, 2024-03-26T06:48:03.207 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_open": 0, 2024-03-26T06:48:03.207 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_stale": 0, 2024-03-26T06:48:03.207 INFO:teuthology.orchestra.run.smithi170.stdout: "total_load": 0, 2024-03-26T06:48:03.207 INFO:teuthology.orchestra.run.smithi170.stdout: "average_load": 0, 2024-03-26T06:48:03.207 INFO:teuthology.orchestra.run.smithi170.stdout: "avg_session_uptime": 2927, 2024-03-26T06:48:03.207 INFO:teuthology.orchestra.run.smithi170.stdout: "mdthresh_evicted": 0 2024-03-26T06:48:03.207 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.207 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-03-26T06:48:03.208 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-03-26T06:48:03.208 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-03-26T06:48:03.208 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:48:03.208 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:48:03.208 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:48:03.208 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:48:03.208 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:48:03.208 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:48:03.209 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:48:03.209 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:48:03.209 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:48:03.209 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:48:03.209 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:48:03.209 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:48:03.209 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:48:03.209 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-03-26T06:48:03.209 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:48:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-03-26T06:48:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:48:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:48:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:48:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:48:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:48:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:48:03.210 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:48:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-03-26T06:48:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:48:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:48:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:48:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-03-26T06:48:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-03-26T06:48:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-03-26T06:48:03.211 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:48:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:48:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:48:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:48:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 242597, 2024-03-26T06:48:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 845, 2024-03-26T06:48:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 0, 2024-03-26T06:48:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 0, 2024-03-26T06:48:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-03-26T06:48:03.212 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-03-26T06:48:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:48:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-03-26T06:48:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:48:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-03-26T06:48:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 8352, 2024-03-26T06:48:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 190, 2024-03-26T06:48:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:48:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:48:03.213 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-03-26T06:48:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-03-26T06:48:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 2759779, 2024-03-26T06:48:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 23704, 2024-03-26T06:48:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-03-26T06:48:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-03-26T06:48:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-03-26T06:48:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-03-26T06:48:03.214 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter": { 2024-03-26T06:48:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: "op_active": 0, 2024-03-26T06:48:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: "op_laggy": 0, 2024-03-26T06:48:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send": 1837, 2024-03-26T06:48:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send_bytes": 26, 2024-03-26T06:48:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: "op_resend": 4, 2024-03-26T06:48:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: "op_reply": 1837, 2024-03-26T06:48:03.215 INFO:teuthology.orchestra.run.smithi170.stdout: "op_latency": { 2024-03-26T06:48:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1837, 2024-03-26T06:48:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 48.090867404, 2024-03-26T06:48:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.026179024 2024-03-26T06:48:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "op_inflight": 0, 2024-03-26T06:48:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "oplen_avg": { 2024-03-26T06:48:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1837, 2024-03-26T06:48:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 1838 2024-03-26T06:48:03.216 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "op": 1837, 2024-03-26T06:48:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "op_r": 1837, 2024-03-26T06:48:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "op_w": 0, 2024-03-26T06:48:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "op_rmw": 0, 2024-03-26T06:48:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "op_pg": 0, 2024-03-26T06:48:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_stat": 1109, 2024-03-26T06:48:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_create": 0, 2024-03-26T06:48:03.217 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_read": 725, 2024-03-26T06:48:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_write": 0, 2024-03-26T06:48:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writefull": 0, 2024-03-26T06:48:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writesame": 0, 2024-03-26T06:48:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_append": 0, 2024-03-26T06:48:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_zero": 0, 2024-03-26T06:48:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_truncate": 0, 2024-03-26T06:48:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_delete": 0, 2024-03-26T06:48:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_mapext": 0, 2024-03-26T06:48:03.218 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_sparse_read": 0, 2024-03-26T06:48:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_clonerange": 0, 2024-03-26T06:48:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_getxattr": 2, 2024-03-26T06:48:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_setxattr": 0, 2024-03-26T06:48:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_cmpxattr": 0, 2024-03-26T06:48:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_rmxattr": 0, 2024-03-26T06:48:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_resetxattrs": 0, 2024-03-26T06:48:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_call": 0, 2024-03-26T06:48:03.219 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_watch": 0, 2024-03-26T06:48:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_notify": 0, 2024-03-26T06:48:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_src_cmpxattr": 0, 2024-03-26T06:48:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls": 0, 2024-03-26T06:48:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls_filter": 0, 2024-03-26T06:48:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_other": 0, 2024-03-26T06:48:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_active": 0, 2024-03-26T06:48:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_send": 0, 2024-03-26T06:48:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_resend": 0, 2024-03-26T06:48:03.220 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_ping": 0, 2024-03-26T06:48:03.221 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_active": 0, 2024-03-26T06:48:03.221 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_send": 0, 2024-03-26T06:48:03.221 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_resend": 0, 2024-03-26T06:48:03.221 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_active": 0, 2024-03-26T06:48:03.221 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_send": 0, 2024-03-26T06:48:03.221 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_resend": 0, 2024-03-26T06:48:03.221 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_active": 0, 2024-03-26T06:48:03.221 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_send": 0, 2024-03-26T06:48:03.221 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_resend": 0, 2024-03-26T06:48:03.222 INFO:teuthology.orchestra.run.smithi170.stdout: "command_active": 0, 2024-03-26T06:48:03.222 INFO:teuthology.orchestra.run.smithi170.stdout: "command_send": 0, 2024-03-26T06:48:03.222 INFO:teuthology.orchestra.run.smithi170.stdout: "command_resend": 0, 2024-03-26T06:48:03.222 INFO:teuthology.orchestra.run.smithi170.stdout: "map_epoch": 98, 2024-03-26T06:48:03.222 INFO:teuthology.orchestra.run.smithi170.stdout: "map_full": 0, 2024-03-26T06:48:03.222 INFO:teuthology.orchestra.run.smithi170.stdout: "map_inc": 8, 2024-03-26T06:48:03.222 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_sessions": 8, 2024-03-26T06:48:03.222 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_open": 8, 2024-03-26T06:48:03.222 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_close": 0, 2024-03-26T06:48:03.223 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_laggy": 0, 2024-03-26T06:48:03.223 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_wr": 0, 2024-03-26T06:48:03.223 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_rd": 2, 2024-03-26T06:48:03.223 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_del": 0 2024-03-26T06:48:03.223 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.223 INFO:teuthology.orchestra.run.smithi170.stdout: "oft": { 2024-03-26T06:48:03.223 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_objs": 0, 2024-03-26T06:48:03.223 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_kv_pairs": 0, 2024-03-26T06:48:03.224 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_updates": 0, 2024-03-26T06:48:03.224 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_removes": 0 2024-03-26T06:48:03.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.224 INFO:teuthology.orchestra.run.smithi170.stdout: "osd-slow-ops": { 2024-03-26T06:48:03.224 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_ops_count": 0 2024-03-26T06:48:03.224 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.224 INFO:teuthology.orchestra.run.smithi170.stdout: "purge_queue": { 2024-03-26T06:48:03.224 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops": 0, 2024-03-26T06:48:03.224 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops_high_water": 0, 2024-03-26T06:48:03.225 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing": 0, 2024-03-26T06:48:03.225 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_high_water": 0, 2024-03-26T06:48:03.225 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed_ops": 0, 2024-03-26T06:48:03.225 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed": 0, 2024-03-26T06:48:03.225 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_item_in_journal": 0 2024-03-26T06:48:03.225 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.225 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:48:03.225 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:03.225 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-26T06:48:03.226 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:03.226 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 2019, 2024-03-26T06:48:03.226 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 67300222, 2024-03-26T06:48:03.226 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:03.226 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 2019, 2024-03-26T06:48:03.226 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-26T06:48:03.226 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-26T06:48:03.226 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 2019, 2024-03-26T06:48:03.227 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 67300222, 2024-03-26T06:48:03.227 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:03.227 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.227 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.227 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.227 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:03.227 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.227 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_bytes": { 2024-03-26T06:48:03.227 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:03.228 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-26T06:48:03.228 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:03.228 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-26T06:48:03.228 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-26T06:48:03.228 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:03.228 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-26T06:48:03.228 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 1837, 2024-03-26T06:48:03.228 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 66614516, 2024-03-26T06:48:03.229 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 1837, 2024-03-26T06:48:03.229 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 66614516, 2024-03-26T06:48:03.229 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:03.229 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.229 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.229 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.229 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:03.229 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.230 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_ops": { 2024-03-26T06:48:03.230 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:03.230 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 1024, 2024-03-26T06:48:03.230 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:03.230 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-26T06:48:03.230 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-26T06:48:03.230 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:03.231 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-26T06:48:03.231 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 1837, 2024-03-26T06:48:03.231 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 1837, 2024-03-26T06:48:03.231 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 1837, 2024-03-26T06:48:03.231 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 1837, 2024-03-26T06:48:03.231 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:03.231 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.231 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.232 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.232 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:03.232 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.232 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle": { 2024-03-26T06:48:03.232 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:03.232 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-26T06:48:03.232 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:03.232 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-26T06:48:03.233 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-26T06:48:03.233 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:03.233 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-26T06:48:03.233 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-26T06:48:03.233 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-26T06:48:03.233 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-03-26T06:48:03.233 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-03-26T06:48:03.233 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:03.233 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.234 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.234 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.234 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:03.234 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:03.234 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle-0x55b0c2a58960": { 2024-03-26T06:48:03.234 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:03.234 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-26T06:48:03.234 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:03.234 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-26T06:48:03.235 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-26T06:48:03.235 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:03.235 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-26T06:48:03.235 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-26T06:48:03.235 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-26T06:48:03.235 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-03-26T06:48:03.235 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-03-26T06:48:03.235 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:03.236 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:03.236 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:03.236 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:03.236 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:03.236 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:03.236 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-26T06:48:03.929 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-26T06:48:03.929 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-26T06:48:03.929 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-26T06:48:03.929 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-26T06:48:03.929 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-26T06:48:03.930 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-26T06:48:03.930 DEBUG:tasks.check_counter:Getting stats from f 2024-03-26T06:48:03.930 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-26T06:48:04.179 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:48:04.607 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:04 smithi170 ceph-mon[36912]: pgmap v327: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-03-26T06:48:04.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:04 smithi038 ceph-mon[32265]: pgmap v327: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-03-26T06:48:04.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:04 smithi121 ceph-mon[36349]: pgmap v327: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 5.2 KiB/s rd, 23 KiB/s wr, 10 op/s 2024-03-26T06:48:06.015 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-26T06:48:06.015 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:48:06.015 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 726, 2024-03-26T06:48:06.015 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 728, 2024-03-26T06:48:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 1386480, 2024-03-26T06:48:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 230932, 2024-03-26T06:48:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 6, 2024-03-26T06:48:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 2, 2024-03-26T06:48:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.070510985, 2024-03-26T06:48:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.020343787, 2024-03-26T06:48:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.020365924, 2024-03-26T06:48:06.016 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.022153338, 2024-03-26T06:48:06.017 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:06.017 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 726, 2024-03-26T06:48:06.017 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.030577529, 2024-03-26T06:48:06.017 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000042117 2024-03-26T06:48:06.017 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.017 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:06.017 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.017 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.017 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.018 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-26T06:48:06.018 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 0 2024-03-26T06:48:06.018 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.018 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:48:06.018 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 869, 2024-03-26T06:48:06.018 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 850, 2024-03-26T06:48:06.018 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 1780522, 2024-03-26T06:48:06.019 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 326937, 2024-03-26T06:48:06.019 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 4, 2024-03-26T06:48:06.019 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 3, 2024-03-26T06:48:06.019 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.111586902, 2024-03-26T06:48:06.019 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.029796736, 2024-03-26T06:48:06.019 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.053896042, 2024-03-26T06:48:06.019 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.032256676, 2024-03-26T06:48:06.019 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:06.019 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 849, 2024-03-26T06:48:06.020 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.034368020, 2024-03-26T06:48:06.020 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000040480 2024-03-26T06:48:06.020 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.020 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:06.020 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.020 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.020 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.020 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.020 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 1627488, 2024-03-26T06:48:06.021 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 129136 2024-03-26T06:48:06.021 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.021 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:48:06.021 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 710, 2024-03-26T06:48:06.021 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 866, 2024-03-26T06:48:06.021 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 239888, 2024-03-26T06:48:06.021 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 664176, 2024-03-26T06:48:06.021 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 6, 2024-03-26T06:48:06.021 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 4, 2024-03-26T06:48:06.022 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.097492705, 2024-03-26T06:48:06.022 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.031816347, 2024-03-26T06:48:06.022 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.027002668, 2024-03-26T06:48:06.022 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.028360524, 2024-03-26T06:48:06.022 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:06.022 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 864, 2024-03-26T06:48:06.022 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.058259897, 2024-03-26T06:48:06.022 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000067430 2024-03-26T06:48:06.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.023 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:06.023 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.023 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.023 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.023 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-26T06:48:06.023 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 440304 2024-03-26T06:48:06.023 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.024 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-03-26T06:48:06.024 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 1, 2024-03-26T06:48:06.024 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-03-26T06:48:06.024 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.024 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-MDSRank": { 2024-03-26T06:48:06.024 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-26T06:48:06.024 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-26T06:48:06.024 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 3408, 2024-03-26T06:48:06.024 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.111504927, 2024-03-26T06:48:06.025 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000032718 2024-03-26T06:48:06.025 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:06.025 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.025 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-PurgeQueue": { 2024-03-26T06:48:06.025 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-26T06:48:06.025 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-26T06:48:06.025 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.025 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.025 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.026 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:06.026 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.026 INFO:teuthology.orchestra.run.smithi170.stdout: "mds": { 2024-03-26T06:48:06.026 INFO:teuthology.orchestra.run.smithi170.stdout: "request": 0, 2024-03-26T06:48:06.026 INFO:teuthology.orchestra.run.smithi170.stdout: "reply": 0, 2024-03-26T06:48:06.026 INFO:teuthology.orchestra.run.smithi170.stdout: "reply_latency": { 2024-03-26T06:48:06.026 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.026 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.027 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.027 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.027 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_reply": 0, 2024-03-26T06:48:06.027 INFO:teuthology.orchestra.run.smithi170.stdout: "forward": 0, 2024-03-26T06:48:06.027 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_complete": 0, 2024-03-26T06:48:06.027 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_keys": 0, 2024-03-26T06:48:06.027 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_commit": 0, 2024-03-26T06:48:06.027 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_split": 0, 2024-03-26T06:48:06.027 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_merge": 0, 2024-03-26T06:48:06.028 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes": 250, 2024-03-26T06:48:06.028 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_top": 2, 2024-03-26T06:48:06.028 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_bottom": 248, 2024-03-26T06:48:06.028 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pin_tail": 0, 2024-03-26T06:48:06.028 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pinned": 248, 2024-03-26T06:48:06.028 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_expired": 0, 2024-03-26T06:48:06.028 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_with_caps": 0, 2024-03-26T06:48:06.028 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": 0, 2024-03-26T06:48:06.028 INFO:teuthology.orchestra.run.smithi170.stdout: "subtrees": 8, 2024-03-26T06:48:06.029 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse": 0, 2024-03-26T06:48:06.029 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_hit": 0, 2024-03-26T06:48:06.029 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_forward": 0, 2024-03-26T06:48:06.029 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_discover": 0, 2024-03-26T06:48:06.029 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_dir_fetch": 0, 2024-03-26T06:48:06.029 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_remote_ino": 0, 2024-03-26T06:48:06.029 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_lock": 0, 2024-03-26T06:48:06.029 INFO:teuthology.orchestra.run.smithi170.stdout: "load_cent": 0, 2024-03-26T06:48:06.030 INFO:teuthology.orchestra.run.smithi170.stdout: "q": 0, 2024-03-26T06:48:06.030 INFO:teuthology.orchestra.run.smithi170.stdout: "exported": 0, 2024-03-26T06:48:06.030 INFO:teuthology.orchestra.run.smithi170.stdout: "exported_inodes": 0, 2024-03-26T06:48:06.030 INFO:teuthology.orchestra.run.smithi170.stdout: "imported": 0, 2024-03-26T06:48:06.030 INFO:teuthology.orchestra.run.smithi170.stdout: "imported_inodes": 0, 2024-03-26T06:48:06.030 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_dir_fetch": 0, 2024-03-26T06:48:06.030 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_backtrace_fetch": 0, 2024-03-26T06:48:06.030 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_peer_discover": 0, 2024-03-26T06:48:06.030 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rfiles": 0, 2024-03-26T06:48:06.031 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rbytes": 0, 2024-03-26T06:48:06.031 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rsnaps": 0, 2024-03-26T06:48:06.031 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_fetch": 0, 2024-03-26T06:48:06.031 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_set_tag": 0, 2024-03-26T06:48:06.031 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_repaired": 0, 2024-03-26T06:48:06.031 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_inotable_repaired": 0, 2024-03-26T06:48:06.031 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_inodes": 0, 2024-03-26T06:48:06.031 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_base_inodes": 0, 2024-03-26T06:48:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dirfrag_rstats": 0, 2024-03-26T06:48:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_file_inodes": 0, 2024-03-26T06:48:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_inode_file_caps": 0, 2024-03-26T06:48:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_revoke": 0, 2024-03-26T06:48:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_grant": 0, 2024-03-26T06:48:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_trunc": 0, 2024-03-26T06:48:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-26T06:48:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-26T06:48:06.032 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps": 0, 2024-03-26T06:48:06.033 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps_dirty": 0, 2024-03-26T06:48:06.033 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_cap_release": 0, 2024-03-26T06:48:06.033 INFO:teuthology.orchestra.run.smithi170.stdout: "process_request_cap_release": 0 2024-03-26T06:48:06.033 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.033 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache": { 2024-03-26T06:48:06.033 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update": 0, 2024-03-26T06:48:06.033 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update_receipt": 0, 2024-03-26T06:48:06.033 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_try_discover": 0, 2024-03-26T06:48:06.033 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_send_discover": 0, 2024-03-26T06:48:06.034 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_handle_discover": 0, 2024-03-26T06:48:06.034 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays": 0, 2024-03-26T06:48:06.034 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_delayed": 0, 2024-03-26T06:48:06.034 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_enqueuing": 0, 2024-03-26T06:48:06.034 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_created": 0, 2024-03-26T06:48:06.034 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_enqueued": 0, 2024-03-26T06:48:06.034 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_reintegrated": 0, 2024-03-26T06:48:06.034 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_migrated": 0, 2024-03-26T06:48:06.035 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_processing": 0, 2024-03-26T06:48:06.035 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_enqueued": 0, 2024-03-26T06:48:06.035 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_prioritized": 0, 2024-03-26T06:48:06.035 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_started": 0, 2024-03-26T06:48:06.035 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_completed": 0, 2024-03-26T06:48:06.035 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_enqueue_scrub": 0, 2024-03-26T06:48:06.035 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_exportdir": 0, 2024-03-26T06:48:06.035 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_flush": 0, 2024-03-26T06:48:06.035 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragmentdir": 0, 2024-03-26T06:48:06.036 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragstats": 0, 2024-03-26T06:48:06.036 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_inodestats": 0 2024-03-26T06:48:06.036 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.036 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log": { 2024-03-26T06:48:06.036 INFO:teuthology.orchestra.run.smithi170.stdout: "evlrg": 0, 2024-03-26T06:48:06.036 INFO:teuthology.orchestra.run.smithi170.stdout: "evadd": 0, 2024-03-26T06:48:06.036 INFO:teuthology.orchestra.run.smithi170.stdout: "evex": 0, 2024-03-26T06:48:06.036 INFO:teuthology.orchestra.run.smithi170.stdout: "evtrm": 0, 2024-03-26T06:48:06.036 INFO:teuthology.orchestra.run.smithi170.stdout: "ev": 264, 2024-03-26T06:48:06.037 INFO:teuthology.orchestra.run.smithi170.stdout: "evexg": 0, 2024-03-26T06:48:06.037 INFO:teuthology.orchestra.run.smithi170.stdout: "evexd": 0, 2024-03-26T06:48:06.037 INFO:teuthology.orchestra.run.smithi170.stdout: "segadd": 0, 2024-03-26T06:48:06.037 INFO:teuthology.orchestra.run.smithi170.stdout: "segex": 0, 2024-03-26T06:48:06.037 INFO:teuthology.orchestra.run.smithi170.stdout: "segtrm": 0, 2024-03-26T06:48:06.037 INFO:teuthology.orchestra.run.smithi170.stdout: "seg": 2, 2024-03-26T06:48:06.037 INFO:teuthology.orchestra.run.smithi170.stdout: "segmjr": 2, 2024-03-26T06:48:06.037 INFO:teuthology.orchestra.run.smithi170.stdout: "segexg": 0, 2024-03-26T06:48:06.037 INFO:teuthology.orchestra.run.smithi170.stdout: "segexd": 0, 2024-03-26T06:48:06.038 INFO:teuthology.orchestra.run.smithi170.stdout: "expos": 4194304, 2024-03-26T06:48:06.038 INFO:teuthology.orchestra.run.smithi170.stdout: "wrpos": 5390237, 2024-03-26T06:48:06.038 INFO:teuthology.orchestra.run.smithi170.stdout: "rdpos": 5389663, 2024-03-26T06:48:06.038 INFO:teuthology.orchestra.run.smithi170.stdout: "jlat": { 2024-03-26T06:48:06.038 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.038 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.038 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.038 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.039 INFO:teuthology.orchestra.run.smithi170.stdout: "replayed": 264 2024-03-26T06:48:06.039 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.039 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mem": { 2024-03-26T06:48:06.039 INFO:teuthology.orchestra.run.smithi170.stdout: "ino": 186, 2024-03-26T06:48:06.039 INFO:teuthology.orchestra.run.smithi170.stdout: "ino+": 186, 2024-03-26T06:48:06.039 INFO:teuthology.orchestra.run.smithi170.stdout: "ino-": 0, 2024-03-26T06:48:06.039 INFO:teuthology.orchestra.run.smithi170.stdout: "dir": 83, 2024-03-26T06:48:06.039 INFO:teuthology.orchestra.run.smithi170.stdout: "dir+": 83, 2024-03-26T06:48:06.039 INFO:teuthology.orchestra.run.smithi170.stdout: "dir-": 0, 2024-03-26T06:48:06.040 INFO:teuthology.orchestra.run.smithi170.stdout: "dn": 250, 2024-03-26T06:48:06.040 INFO:teuthology.orchestra.run.smithi170.stdout: "dn+": 252, 2024-03-26T06:48:06.040 INFO:teuthology.orchestra.run.smithi170.stdout: "dn-": 2, 2024-03-26T06:48:06.040 INFO:teuthology.orchestra.run.smithi170.stdout: "cap": 0, 2024-03-26T06:48:06.040 INFO:teuthology.orchestra.run.smithi170.stdout: "cap+": 0, 2024-03-26T06:48:06.040 INFO:teuthology.orchestra.run.smithi170.stdout: "cap-": 0, 2024-03-26T06:48:06.040 INFO:teuthology.orchestra.run.smithi170.stdout: "rss": 48532, 2024-03-26T06:48:06.040 INFO:teuthology.orchestra.run.smithi170.stdout: "heap": 215420 2024-03-26T06:48:06.040 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.041 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_server": { 2024-03-26T06:48:06.041 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_client_request": 0, 2024-03-26T06:48:06.041 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_server_request": 0, 2024-03-26T06:48:06.041 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_request": 0, 2024-03-26T06:48:06.041 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_session": 0, 2024-03-26T06:48:06.041 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_peer_request": 0, 2024-03-26T06:48:06.041 INFO:teuthology.orchestra.run.smithi170.stdout: "req_create_latency": { 2024-03-26T06:48:06.041 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.041 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.042 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.042 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getattr_latency": { 2024-03-26T06:48:06.042 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.042 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.042 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.042 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.042 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getfilelock_latency": { 2024-03-26T06:48:06.043 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.043 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.043 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.043 INFO:teuthology.orchestra.run.smithi170.stdout: "req_link_latency": { 2024-03-26T06:48:06.043 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.043 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.043 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.043 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.044 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookup_latency": { 2024-03-26T06:48:06.044 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.044 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.044 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.044 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.044 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookuphash_latency": { 2024-03-26T06:48:06.044 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.044 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.044 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.045 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupino_latency": { 2024-03-26T06:48:06.045 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.045 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.045 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.045 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.045 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupname_latency": { 2024-03-26T06:48:06.045 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.045 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.046 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.046 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupparent_latency": { 2024-03-26T06:48:06.046 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.046 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.046 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.046 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.046 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupsnap_latency": { 2024-03-26T06:48:06.047 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.047 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.047 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.047 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lssnap_latency": { 2024-03-26T06:48:06.047 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.047 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.047 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.047 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.048 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mkdir_latency": { 2024-03-26T06:48:06.048 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.048 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.049 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.049 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mknod_latency": { 2024-03-26T06:48:06.049 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.049 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.049 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.049 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.049 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mksnap_latency": { 2024-03-26T06:48:06.050 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.050 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.050 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.050 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.050 INFO:teuthology.orchestra.run.smithi170.stdout: "req_open_latency": { 2024-03-26T06:48:06.050 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.051 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.051 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.051 INFO:teuthology.orchestra.run.smithi170.stdout: "req_readdir_latency": { 2024-03-26T06:48:06.051 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.051 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.051 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.051 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.052 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rename_latency": { 2024-03-26T06:48:06.052 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.052 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.052 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.052 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.052 INFO:teuthology.orchestra.run.smithi170.stdout: "req_renamesnap_latency": { 2024-03-26T06:48:06.052 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.052 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.053 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.053 INFO:teuthology.orchestra.run.smithi170.stdout: "req_snapdiff_latency": { 2024-03-26T06:48:06.053 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.053 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.053 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.053 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.053 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmdir_latency": { 2024-03-26T06:48:06.053 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.054 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.054 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.054 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmsnap_latency": { 2024-03-26T06:48:06.054 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.054 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.054 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.054 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.055 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmxattr_latency": { 2024-03-26T06:48:06.055 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.055 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.055 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.055 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.055 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setattr_latency": { 2024-03-26T06:48:06.055 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.055 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.055 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.056 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.056 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setdirlayout_latency": { 2024-03-26T06:48:06.056 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.056 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.056 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.056 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.056 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setfilelock_latency": { 2024-03-26T06:48:06.056 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.056 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.057 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.057 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setlayout_latency": { 2024-03-26T06:48:06.057 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.057 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.057 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.057 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.057 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setxattr_latency": { 2024-03-26T06:48:06.058 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.058 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.058 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.058 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.058 INFO:teuthology.orchestra.run.smithi170.stdout: "req_symlink_latency": { 2024-03-26T06:48:06.058 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.058 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.058 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.059 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.059 INFO:teuthology.orchestra.run.smithi170.stdout: "req_unlink_latency": { 2024-03-26T06:48:06.059 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.059 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.059 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.059 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.059 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_revoke_eviction": 0, 2024-03-26T06:48:06.059 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_acquisition_throttle": 0, 2024-03-26T06:48:06.060 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getvxattr_latency": { 2024-03-26T06:48:06.060 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.060 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.060 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.060 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:06.060 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.060 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessions": { 2024-03-26T06:48:06.060 INFO:teuthology.orchestra.run.smithi170.stdout: "session_count": 2, 2024-03-26T06:48:06.061 INFO:teuthology.orchestra.run.smithi170.stdout: "session_add": 5, 2024-03-26T06:48:06.061 INFO:teuthology.orchestra.run.smithi170.stdout: "session_remove": 3, 2024-03-26T06:48:06.061 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_open": 3, 2024-03-26T06:48:06.061 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_stale": 0, 2024-03-26T06:48:06.061 INFO:teuthology.orchestra.run.smithi170.stdout: "total_load": 0, 2024-03-26T06:48:06.061 INFO:teuthology.orchestra.run.smithi170.stdout: "average_load": 0, 2024-03-26T06:48:06.061 INFO:teuthology.orchestra.run.smithi170.stdout: "avg_session_uptime": 2912, 2024-03-26T06:48:06.061 INFO:teuthology.orchestra.run.smithi170.stdout: "mdthresh_evicted": 0 2024-03-26T06:48:06.062 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.062 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-03-26T06:48:06.062 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-03-26T06:48:06.062 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-03-26T06:48:06.062 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:48:06.062 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:48:06.063 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:48:06.063 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:48:06.063 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:48:06.063 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:48:06.063 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:48:06.063 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:48:06.063 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:48:06.064 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:48:06.064 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:48:06.064 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:48:06.064 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:48:06.064 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-03-26T06:48:06.064 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:48:06.064 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-03-26T06:48:06.065 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:48:06.065 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:48:06.065 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:48:06.065 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:48:06.065 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:48:06.065 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:48:06.065 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:48:06.066 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-03-26T06:48:06.066 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:48:06.066 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:48:06.066 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:48:06.066 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-03-26T06:48:06.066 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-03-26T06:48:06.066 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-03-26T06:48:06.066 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:48:06.067 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:48:06.067 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:48:06.067 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:48:06.067 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 198636, 2024-03-26T06:48:06.067 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 300, 2024-03-26T06:48:06.067 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 0, 2024-03-26T06:48:06.067 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 0, 2024-03-26T06:48:06.067 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-03-26T06:48:06.067 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-03-26T06:48:06.068 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:48:06.068 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-03-26T06:48:06.068 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:48:06.068 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-03-26T06:48:06.068 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 8352, 2024-03-26T06:48:06.068 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 190, 2024-03-26T06:48:06.068 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:48:06.068 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:48:06.069 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-03-26T06:48:06.069 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-03-26T06:48:06.069 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 783325, 2024-03-26T06:48:06.069 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 15603, 2024-03-26T06:48:06.069 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-03-26T06:48:06.069 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-03-26T06:48:06.069 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-03-26T06:48:06.069 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-03-26T06:48:06.069 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.070 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter": { 2024-03-26T06:48:06.070 INFO:teuthology.orchestra.run.smithi170.stdout: "op_active": 0, 2024-03-26T06:48:06.070 INFO:teuthology.orchestra.run.smithi170.stdout: "op_laggy": 0, 2024-03-26T06:48:06.070 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send": 2052, 2024-03-26T06:48:06.070 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send_bytes": 21, 2024-03-26T06:48:06.070 INFO:teuthology.orchestra.run.smithi170.stdout: "op_resend": 3, 2024-03-26T06:48:06.070 INFO:teuthology.orchestra.run.smithi170.stdout: "op_reply": 2052, 2024-03-26T06:48:06.070 INFO:teuthology.orchestra.run.smithi170.stdout: "op_latency": { 2024-03-26T06:48:06.071 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2052, 2024-03-26T06:48:06.071 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 51.450787818, 2024-03-26T06:48:06.071 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.025073483 2024-03-26T06:48:06.071 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.071 INFO:teuthology.orchestra.run.smithi170.stdout: "op_inflight": 0, 2024-03-26T06:48:06.071 INFO:teuthology.orchestra.run.smithi170.stdout: "oplen_avg": { 2024-03-26T06:48:06.071 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2052, 2024-03-26T06:48:06.071 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 2053 2024-03-26T06:48:06.071 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.072 INFO:teuthology.orchestra.run.smithi170.stdout: "op": 2052, 2024-03-26T06:48:06.072 INFO:teuthology.orchestra.run.smithi170.stdout: "op_r": 2052, 2024-03-26T06:48:06.072 INFO:teuthology.orchestra.run.smithi170.stdout: "op_w": 0, 2024-03-26T06:48:06.072 INFO:teuthology.orchestra.run.smithi170.stdout: "op_rmw": 0, 2024-03-26T06:48:06.072 INFO:teuthology.orchestra.run.smithi170.stdout: "op_pg": 0, 2024-03-26T06:48:06.072 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_stat": 1324, 2024-03-26T06:48:06.072 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_create": 0, 2024-03-26T06:48:06.072 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_read": 726, 2024-03-26T06:48:06.073 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_write": 0, 2024-03-26T06:48:06.073 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writefull": 0, 2024-03-26T06:48:06.073 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writesame": 0, 2024-03-26T06:48:06.073 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_append": 0, 2024-03-26T06:48:06.073 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_zero": 0, 2024-03-26T06:48:06.073 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_truncate": 0, 2024-03-26T06:48:06.073 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_delete": 0, 2024-03-26T06:48:06.073 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_mapext": 0, 2024-03-26T06:48:06.073 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_sparse_read": 0, 2024-03-26T06:48:06.074 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_clonerange": 0, 2024-03-26T06:48:06.074 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_getxattr": 1, 2024-03-26T06:48:06.074 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_setxattr": 0, 2024-03-26T06:48:06.074 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_cmpxattr": 0, 2024-03-26T06:48:06.074 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_rmxattr": 0, 2024-03-26T06:48:06.074 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_resetxattrs": 0, 2024-03-26T06:48:06.074 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_call": 0, 2024-03-26T06:48:06.074 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_watch": 0, 2024-03-26T06:48:06.074 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_notify": 0, 2024-03-26T06:48:06.075 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_src_cmpxattr": 0, 2024-03-26T06:48:06.075 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls": 0, 2024-03-26T06:48:06.075 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls_filter": 0, 2024-03-26T06:48:06.075 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_other": 0, 2024-03-26T06:48:06.075 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_active": 0, 2024-03-26T06:48:06.075 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_send": 0, 2024-03-26T06:48:06.075 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_resend": 0, 2024-03-26T06:48:06.075 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_ping": 0, 2024-03-26T06:48:06.076 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_active": 0, 2024-03-26T06:48:06.076 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_send": 0, 2024-03-26T06:48:06.076 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_resend": 0, 2024-03-26T06:48:06.076 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_active": 0, 2024-03-26T06:48:06.076 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_send": 0, 2024-03-26T06:48:06.076 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_resend": 0, 2024-03-26T06:48:06.076 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_active": 0, 2024-03-26T06:48:06.076 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_send": 0, 2024-03-26T06:48:06.076 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_resend": 0, 2024-03-26T06:48:06.077 INFO:teuthology.orchestra.run.smithi170.stdout: "command_active": 0, 2024-03-26T06:48:06.077 INFO:teuthology.orchestra.run.smithi170.stdout: "command_send": 0, 2024-03-26T06:48:06.077 INFO:teuthology.orchestra.run.smithi170.stdout: "command_resend": 0, 2024-03-26T06:48:06.077 INFO:teuthology.orchestra.run.smithi170.stdout: "map_epoch": 98, 2024-03-26T06:48:06.077 INFO:teuthology.orchestra.run.smithi170.stdout: "map_full": 0, 2024-03-26T06:48:06.077 INFO:teuthology.orchestra.run.smithi170.stdout: "map_inc": 12, 2024-03-26T06:48:06.077 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_sessions": 5, 2024-03-26T06:48:06.077 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_open": 5, 2024-03-26T06:48:06.077 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_close": 0, 2024-03-26T06:48:06.078 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_laggy": 0, 2024-03-26T06:48:06.078 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_wr": 0, 2024-03-26T06:48:06.078 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_rd": 2, 2024-03-26T06:48:06.078 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_del": 0 2024-03-26T06:48:06.078 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.078 INFO:teuthology.orchestra.run.smithi170.stdout: "oft": { 2024-03-26T06:48:06.078 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_objs": 0, 2024-03-26T06:48:06.078 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_kv_pairs": 0, 2024-03-26T06:48:06.079 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_updates": 0, 2024-03-26T06:48:06.079 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_removes": 0 2024-03-26T06:48:06.079 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.079 INFO:teuthology.orchestra.run.smithi170.stdout: "osd-slow-ops": { 2024-03-26T06:48:06.079 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_ops_count": 0 2024-03-26T06:48:06.079 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.079 INFO:teuthology.orchestra.run.smithi170.stdout: "purge_queue": { 2024-03-26T06:48:06.079 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops": 0, 2024-03-26T06:48:06.079 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops_high_water": 0, 2024-03-26T06:48:06.080 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing": 0, 2024-03-26T06:48:06.080 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_high_water": 0, 2024-03-26T06:48:06.080 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed_ops": 0, 2024-03-26T06:48:06.080 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed": 0, 2024-03-26T06:48:06.080 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_item_in_journal": 0 2024-03-26T06:48:06.080 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.080 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:48:06.080 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:06.081 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-26T06:48:06.081 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:06.081 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 2301, 2024-03-26T06:48:06.081 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 3184209, 2024-03-26T06:48:06.081 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:06.081 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 2301, 2024-03-26T06:48:06.081 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-26T06:48:06.081 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-26T06:48:06.081 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 2301, 2024-03-26T06:48:06.082 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 3184209, 2024-03-26T06:48:06.082 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:06.082 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.082 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.082 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.082 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:06.082 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.083 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_bytes": { 2024-03-26T06:48:06.083 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:06.083 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-26T06:48:06.083 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:06.083 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-26T06:48:06.083 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-26T06:48:06.083 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:06.083 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-26T06:48:06.083 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 2052, 2024-03-26T06:48:06.084 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 1195938, 2024-03-26T06:48:06.084 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 2052, 2024-03-26T06:48:06.084 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 1195938, 2024-03-26T06:48:06.084 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:06.084 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.084 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.084 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.084 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:06.085 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.085 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_ops": { 2024-03-26T06:48:06.085 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:06.085 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 1024, 2024-03-26T06:48:06.085 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:06.085 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-26T06:48:06.085 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-26T06:48:06.085 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:06.085 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-26T06:48:06.086 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 2052, 2024-03-26T06:48:06.086 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 2052, 2024-03-26T06:48:06.086 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 2052, 2024-03-26T06:48:06.086 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 2052, 2024-03-26T06:48:06.086 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:06.086 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.086 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.086 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.087 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:06.087 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.087 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle": { 2024-03-26T06:48:06.087 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:06.087 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-26T06:48:06.087 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:06.087 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-26T06:48:06.087 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-26T06:48:06.087 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:06.088 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-26T06:48:06.088 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-26T06:48:06.088 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-26T06:48:06.088 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-03-26T06:48:06.088 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-03-26T06:48:06.088 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:06.088 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.088 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.088 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.089 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:06.089 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:06.089 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle-0x55a8d7e32960": { 2024-03-26T06:48:06.089 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:06.089 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-26T06:48:06.089 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:06.089 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-26T06:48:06.089 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-26T06:48:06.089 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:06.090 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-26T06:48:06.090 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-26T06:48:06.090 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-26T06:48:06.090 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-03-26T06:48:06.090 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-03-26T06:48:06.090 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:06.090 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:06.090 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:06.091 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:06.091 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:06.091 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:06.091 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-26T06:48:06.794 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:06 smithi170 ceph-mon[36912]: pgmap v328: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 23 KiB/s wr, 7 op/s 2024-03-26T06:48:06.824 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-26T06:48:06.824 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-26T06:48:06.825 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-26T06:48:06.825 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-26T06:48:06.825 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-26T06:48:06.825 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-26T06:48:06.825 DEBUG:tasks.check_counter:Getting stats from i 2024-03-26T06:48:06.825 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-26T06:48:06.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:06 smithi038 ceph-mon[32265]: pgmap v328: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 23 KiB/s wr, 7 op/s 2024-03-26T06:48:06.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:06 smithi121 ceph-mon[36349]: pgmap v328: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 3.9 KiB/s rd, 23 KiB/s wr, 7 op/s 2024-03-26T06:48:07.067 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:48:08.695 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-26T06:48:08.696 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:48:08.696 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 327, 2024-03-26T06:48:08.696 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 343, 2024-03-26T06:48:08.696 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 1652569, 2024-03-26T06:48:08.696 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 227120, 2024-03-26T06:48:08.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 14, 2024-03-26T06:48:08.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 8, 2024-03-26T06:48:08.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.056234358, 2024-03-26T06:48:08.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.016905393, 2024-03-26T06:48:08.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.042572427, 2024-03-26T06:48:08.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.006750558, 2024-03-26T06:48:08.697 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:08.697 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 339, 2024-03-26T06:48:08.698 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.022266043, 2024-03-26T06:48:08.698 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000065681 2024-03-26T06:48:08.698 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.698 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:08.698 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 75, 2024-03-26T06:48:08.698 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000012799, 2024-03-26T06:48:08.698 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000170 2024-03-26T06:48:08.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 1646656, 2024-03-26T06:48:08.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 149040 2024-03-26T06:48:08.699 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.699 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:48:08.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 913, 2024-03-26T06:48:08.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 856, 2024-03-26T06:48:08.699 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 280558, 2024-03-26T06:48:08.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 814247, 2024-03-26T06:48:08.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 9, 2024-03-26T06:48:08.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 5, 2024-03-26T06:48:08.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.086942363, 2024-03-26T06:48:08.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.035024843, 2024-03-26T06:48:08.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.073781149, 2024-03-26T06:48:08.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.004955240, 2024-03-26T06:48:08.700 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:08.701 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 857, 2024-03-26T06:48:08.701 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.067264892, 2024-03-26T06:48:08.701 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000078488 2024-03-26T06:48:08.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.701 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:08.701 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 1517, 2024-03-26T06:48:08.701 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000354755, 2024-03-26T06:48:08.701 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000233 2024-03-26T06:48:08.701 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.702 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-26T06:48:08.702 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 443504 2024-03-26T06:48:08.702 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.702 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:48:08.702 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 58, 2024-03-26T06:48:08.702 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 89, 2024-03-26T06:48:08.702 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 14145, 2024-03-26T06:48:08.703 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 28162, 2024-03-26T06:48:08.703 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 7, 2024-03-26T06:48:08.703 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 5, 2024-03-26T06:48:08.703 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.011639900, 2024-03-26T06:48:08.703 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.004018923, 2024-03-26T06:48:08.703 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.007023043, 2024-03-26T06:48:08.703 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.000583221, 2024-03-26T06:48:08.703 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:08.703 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 87, 2024-03-26T06:48:08.704 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.014539279, 2024-03-26T06:48:08.704 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000167118 2024-03-26T06:48:08.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.704 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:08.704 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 61, 2024-03-26T06:48:08.704 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000041367, 2024-03-26T06:48:08.704 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000678 2024-03-26T06:48:08.704 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.705 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-26T06:48:08.705 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 0 2024-03-26T06:48:08.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.705 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-03-26T06:48:08.705 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 1, 2024-03-26T06:48:08.705 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-03-26T06:48:08.705 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.705 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-MDSRank": { 2024-03-26T06:48:08.706 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-26T06:48:08.706 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-26T06:48:08.706 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 60, 2024-03-26T06:48:08.706 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.002526639, 2024-03-26T06:48:08.706 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000042110 2024-03-26T06:48:08.706 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:08.707 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.707 INFO:teuthology.orchestra.run.smithi170.stdout: "finisher-PurgeQueue": { 2024-03-26T06:48:08.707 INFO:teuthology.orchestra.run.smithi170.stdout: "queue_len": 0, 2024-03-26T06:48:08.707 INFO:teuthology.orchestra.run.smithi170.stdout: "complete_latency": { 2024-03-26T06:48:08.707 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2, 2024-03-26T06:48:08.707 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000011713, 2024-03-26T06:48:08.707 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000005856 2024-03-26T06:48:08.707 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:08.708 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.708 INFO:teuthology.orchestra.run.smithi170.stdout: "mds": { 2024-03-26T06:48:08.708 INFO:teuthology.orchestra.run.smithi170.stdout: "request": 31, 2024-03-26T06:48:08.708 INFO:teuthology.orchestra.run.smithi170.stdout: "reply": 8, 2024-03-26T06:48:08.708 INFO:teuthology.orchestra.run.smithi170.stdout: "reply_latency": { 2024-03-26T06:48:08.709 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 8, 2024-03-26T06:48:08.709 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.002828856, 2024-03-26T06:48:08.709 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000353607 2024-03-26T06:48:08.709 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.709 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_reply": 0, 2024-03-26T06:48:08.710 INFO:teuthology.orchestra.run.smithi170.stdout: "forward": 23, 2024-03-26T06:48:08.710 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_complete": 0, 2024-03-26T06:48:08.710 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_fetch_keys": 0, 2024-03-26T06:48:08.710 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_commit": 11, 2024-03-26T06:48:08.711 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_split": 0, 2024-03-26T06:48:08.711 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_merge": 0, 2024-03-26T06:48:08.711 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes": 25, 2024-03-26T06:48:08.711 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_top": 0, 2024-03-26T06:48:08.712 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_bottom": 25, 2024-03-26T06:48:08.712 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pin_tail": 0, 2024-03-26T06:48:08.712 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_pinned": 25, 2024-03-26T06:48:08.712 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_expired": 1, 2024-03-26T06:48:08.713 INFO:teuthology.orchestra.run.smithi170.stdout: "inodes_with_caps": 5, 2024-03-26T06:48:08.713 INFO:teuthology.orchestra.run.smithi170.stdout: "caps": 5, 2024-03-26T06:48:08.713 INFO:teuthology.orchestra.run.smithi170.stdout: "subtrees": 5, 2024-03-26T06:48:08.713 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse": 40, 2024-03-26T06:48:08.714 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_hit": 12, 2024-03-26T06:48:08.714 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_forward": 20, 2024-03-26T06:48:08.714 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_discover": 5, 2024-03-26T06:48:08.714 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_dir_fetch": 0, 2024-03-26T06:48:08.715 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_remote_ino": 0, 2024-03-26T06:48:08.715 INFO:teuthology.orchestra.run.smithi170.stdout: "traverse_lock": 0, 2024-03-26T06:48:08.715 INFO:teuthology.orchestra.run.smithi170.stdout: "load_cent": 0, 2024-03-26T06:48:08.715 INFO:teuthology.orchestra.run.smithi170.stdout: "q": 0, 2024-03-26T06:48:08.715 INFO:teuthology.orchestra.run.smithi170.stdout: "exported": 0, 2024-03-26T06:48:08.716 INFO:teuthology.orchestra.run.smithi170.stdout: "exported_inodes": 0, 2024-03-26T06:48:08.716 INFO:teuthology.orchestra.run.smithi170.stdout: "imported": 2, 2024-03-26T06:48:08.716 INFO:teuthology.orchestra.run.smithi170.stdout: "imported_inodes": 4, 2024-03-26T06:48:08.716 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_dir_fetch": 0, 2024-03-26T06:48:08.717 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_backtrace_fetch": 0, 2024-03-26T06:48:08.717 INFO:teuthology.orchestra.run.smithi170.stdout: "openino_peer_discover": 1, 2024-03-26T06:48:08.717 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rfiles": 324, 2024-03-26T06:48:08.717 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rbytes": 13424234229, 2024-03-26T06:48:08.718 INFO:teuthology.orchestra.run.smithi170.stdout: "root_rsnaps": 5, 2024-03-26T06:48:08.718 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_fetch": 0, 2024-03-26T06:48:08.718 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_set_tag": 0, 2024-03-26T06:48:08.719 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_backtrace_repaired": 0, 2024-03-26T06:48:08.719 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_inotable_repaired": 0, 2024-03-26T06:48:08.719 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_inodes": 0, 2024-03-26T06:48:08.719 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dir_base_inodes": 0, 2024-03-26T06:48:08.720 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_dirfrag_rstats": 0, 2024-03-26T06:48:08.720 INFO:teuthology.orchestra.run.smithi170.stdout: "scrub_file_inodes": 0, 2024-03-26T06:48:08.720 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_inode_file_caps": 0, 2024-03-26T06:48:08.720 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_revoke": 0, 2024-03-26T06:48:08.720 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_grant": 0, 2024-03-26T06:48:08.720 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_trunc": 0, 2024-03-26T06:48:08.720 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-26T06:48:08.721 INFO:teuthology.orchestra.run.smithi170.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-26T06:48:08.721 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps": 4, 2024-03-26T06:48:08.721 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_caps_dirty": 0, 2024-03-26T06:48:08.721 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_cap_release": 0, 2024-03-26T06:48:08.721 INFO:teuthology.orchestra.run.smithi170.stdout: "process_request_cap_release": 0 2024-03-26T06:48:08.721 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.721 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_cache": { 2024-03-26T06:48:08.721 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update": 0, 2024-03-26T06:48:08.722 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_update_receipt": 0, 2024-03-26T06:48:08.722 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_try_discover": 0, 2024-03-26T06:48:08.722 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_send_discover": 5, 2024-03-26T06:48:08.722 INFO:teuthology.orchestra.run.smithi170.stdout: "dir_handle_discover": 2, 2024-03-26T06:48:08.722 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays": 0, 2024-03-26T06:48:08.722 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_delayed": 0, 2024-03-26T06:48:08.723 INFO:teuthology.orchestra.run.smithi170.stdout: "num_strays_enqueuing": 0, 2024-03-26T06:48:08.723 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_created": 0, 2024-03-26T06:48:08.723 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_enqueued": 0, 2024-03-26T06:48:08.723 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_reintegrated": 0, 2024-03-26T06:48:08.723 INFO:teuthology.orchestra.run.smithi170.stdout: "strays_migrated": 0, 2024-03-26T06:48:08.723 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_processing": 0, 2024-03-26T06:48:08.723 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_enqueued": 0, 2024-03-26T06:48:08.724 INFO:teuthology.orchestra.run.smithi170.stdout: "num_recovering_prioritized": 0, 2024-03-26T06:48:08.724 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_started": 0, 2024-03-26T06:48:08.724 INFO:teuthology.orchestra.run.smithi170.stdout: "recovery_completed": 0, 2024-03-26T06:48:08.724 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_enqueue_scrub": 0, 2024-03-26T06:48:08.724 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_exportdir": 0, 2024-03-26T06:48:08.724 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_flush": 0, 2024-03-26T06:48:08.724 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragmentdir": 0, 2024-03-26T06:48:08.724 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_fragstats": 0, 2024-03-26T06:48:08.725 INFO:teuthology.orchestra.run.smithi170.stdout: "ireq_inodestats": 0 2024-03-26T06:48:08.725 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.725 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_log": { 2024-03-26T06:48:08.725 INFO:teuthology.orchestra.run.smithi170.stdout: "evlrg": 0, 2024-03-26T06:48:08.725 INFO:teuthology.orchestra.run.smithi170.stdout: "evadd": 14, 2024-03-26T06:48:08.725 INFO:teuthology.orchestra.run.smithi170.stdout: "evex": 0, 2024-03-26T06:48:08.725 INFO:teuthology.orchestra.run.smithi170.stdout: "evtrm": 0, 2024-03-26T06:48:08.725 INFO:teuthology.orchestra.run.smithi170.stdout: "ev": 14, 2024-03-26T06:48:08.726 INFO:teuthology.orchestra.run.smithi170.stdout: "evexg": 0, 2024-03-26T06:48:08.726 INFO:teuthology.orchestra.run.smithi170.stdout: "evexd": 0, 2024-03-26T06:48:08.726 INFO:teuthology.orchestra.run.smithi170.stdout: "segadd": 2, 2024-03-26T06:48:08.726 INFO:teuthology.orchestra.run.smithi170.stdout: "segex": 0, 2024-03-26T06:48:08.726 INFO:teuthology.orchestra.run.smithi170.stdout: "segtrm": 0, 2024-03-26T06:48:08.726 INFO:teuthology.orchestra.run.smithi170.stdout: "seg": 2, 2024-03-26T06:48:08.726 INFO:teuthology.orchestra.run.smithi170.stdout: "segmjr": 2, 2024-03-26T06:48:08.727 INFO:teuthology.orchestra.run.smithi170.stdout: "segexg": 0, 2024-03-26T06:48:08.727 INFO:teuthology.orchestra.run.smithi170.stdout: "segexd": 0, 2024-03-26T06:48:08.727 INFO:teuthology.orchestra.run.smithi170.stdout: "expos": 4194304, 2024-03-26T06:48:08.727 INFO:teuthology.orchestra.run.smithi170.stdout: "wrpos": 4224893, 2024-03-26T06:48:08.727 INFO:teuthology.orchestra.run.smithi170.stdout: "rdpos": 0, 2024-03-26T06:48:08.727 INFO:teuthology.orchestra.run.smithi170.stdout: "jlat": { 2024-03-26T06:48:08.727 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 10, 2024-03-26T06:48:08.727 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.018537903, 2024-03-26T06:48:08.728 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.001853790 2024-03-26T06:48:08.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.728 INFO:teuthology.orchestra.run.smithi170.stdout: "replayed": 0 2024-03-26T06:48:08.728 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.728 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_mem": { 2024-03-26T06:48:08.728 INFO:teuthology.orchestra.run.smithi170.stdout: "ino": 26, 2024-03-26T06:48:08.728 INFO:teuthology.orchestra.run.smithi170.stdout: "ino+": 26, 2024-03-26T06:48:08.728 INFO:teuthology.orchestra.run.smithi170.stdout: "ino-": 0, 2024-03-26T06:48:08.729 INFO:teuthology.orchestra.run.smithi170.stdout: "dir": 30, 2024-03-26T06:48:08.729 INFO:teuthology.orchestra.run.smithi170.stdout: "dir+": 32, 2024-03-26T06:48:08.729 INFO:teuthology.orchestra.run.smithi170.stdout: "dir-": 2, 2024-03-26T06:48:08.729 INFO:teuthology.orchestra.run.smithi170.stdout: "dn": 25, 2024-03-26T06:48:08.729 INFO:teuthology.orchestra.run.smithi170.stdout: "dn+": 26, 2024-03-26T06:48:08.729 INFO:teuthology.orchestra.run.smithi170.stdout: "dn-": 1, 2024-03-26T06:48:08.729 INFO:teuthology.orchestra.run.smithi170.stdout: "cap": 5, 2024-03-26T06:48:08.729 INFO:teuthology.orchestra.run.smithi170.stdout: "cap+": 7, 2024-03-26T06:48:08.729 INFO:teuthology.orchestra.run.smithi170.stdout: "cap-": 2, 2024-03-26T06:48:08.730 INFO:teuthology.orchestra.run.smithi170.stdout: "rss": 53040, 2024-03-26T06:48:08.730 INFO:teuthology.orchestra.run.smithi170.stdout: "heap": 207228 2024-03-26T06:48:08.730 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.730 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_server": { 2024-03-26T06:48:08.730 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_client_request": 33, 2024-03-26T06:48:08.730 INFO:teuthology.orchestra.run.smithi170.stdout: "dispatch_server_request": 0, 2024-03-26T06:48:08.730 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_request": 31, 2024-03-26T06:48:08.731 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_client_session": 22, 2024-03-26T06:48:08.731 INFO:teuthology.orchestra.run.smithi170.stdout: "handle_peer_request": 0, 2024-03-26T06:48:08.731 INFO:teuthology.orchestra.run.smithi170.stdout: "req_create_latency": { 2024-03-26T06:48:08.731 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.731 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.731 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.731 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.731 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getattr_latency": { 2024-03-26T06:48:08.731 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 4, 2024-03-26T06:48:08.732 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.001123818, 2024-03-26T06:48:08.732 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000280954 2024-03-26T06:48:08.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.732 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getfilelock_latency": { 2024-03-26T06:48:08.732 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.732 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.732 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.732 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.732 INFO:teuthology.orchestra.run.smithi170.stdout: "req_link_latency": { 2024-03-26T06:48:08.733 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.733 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.733 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.733 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.733 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookup_latency": { 2024-03-26T06:48:08.733 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2, 2024-03-26T06:48:08.733 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000858534, 2024-03-26T06:48:08.734 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000429267 2024-03-26T06:48:08.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.734 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookuphash_latency": { 2024-03-26T06:48:08.734 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.734 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.734 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.734 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.734 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupino_latency": { 2024-03-26T06:48:08.735 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.735 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.735 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.735 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.735 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupname_latency": { 2024-03-26T06:48:08.735 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.735 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.735 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.736 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupparent_latency": { 2024-03-26T06:48:08.736 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.736 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.736 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.736 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.736 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lookupsnap_latency": { 2024-03-26T06:48:08.736 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.737 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.737 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.737 INFO:teuthology.orchestra.run.smithi170.stdout: "req_lssnap_latency": { 2024-03-26T06:48:08.737 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.737 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.737 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.737 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.738 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mkdir_latency": { 2024-03-26T06:48:08.738 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.738 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.739 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.739 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.739 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mknod_latency": { 2024-03-26T06:48:08.739 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.739 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.739 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.740 INFO:teuthology.orchestra.run.smithi170.stdout: "req_mksnap_latency": { 2024-03-26T06:48:08.740 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.740 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.740 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.740 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.740 INFO:teuthology.orchestra.run.smithi170.stdout: "req_open_latency": { 2024-03-26T06:48:08.740 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.741 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.741 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.741 INFO:teuthology.orchestra.run.smithi170.stdout: "req_readdir_latency": { 2024-03-26T06:48:08.741 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 2, 2024-03-26T06:48:08.741 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000846504, 2024-03-26T06:48:08.741 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000423252 2024-03-26T06:48:08.741 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.742 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rename_latency": { 2024-03-26T06:48:08.742 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.742 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.742 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.742 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.742 INFO:teuthology.orchestra.run.smithi170.stdout: "req_renamesnap_latency": { 2024-03-26T06:48:08.742 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.743 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.743 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.743 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.743 INFO:teuthology.orchestra.run.smithi170.stdout: "req_snapdiff_latency": { 2024-03-26T06:48:08.743 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.743 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.743 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.743 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.744 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmdir_latency": { 2024-03-26T06:48:08.744 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.744 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.744 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.744 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.744 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmsnap_latency": { 2024-03-26T06:48:08.744 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.744 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.744 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.745 INFO:teuthology.orchestra.run.smithi170.stdout: "req_rmxattr_latency": { 2024-03-26T06:48:08.745 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.745 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.745 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.745 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.745 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setattr_latency": { 2024-03-26T06:48:08.746 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.746 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.746 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.746 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.746 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setdirlayout_latency": { 2024-03-26T06:48:08.746 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.746 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.747 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.747 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setfilelock_latency": { 2024-03-26T06:48:08.747 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.747 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.747 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.747 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.747 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setlayout_latency": { 2024-03-26T06:48:08.747 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.748 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.748 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.748 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.748 INFO:teuthology.orchestra.run.smithi170.stdout: "req_setxattr_latency": { 2024-03-26T06:48:08.748 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.748 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.748 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.749 INFO:teuthology.orchestra.run.smithi170.stdout: "req_symlink_latency": { 2024-03-26T06:48:08.749 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.749 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.749 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.749 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.749 INFO:teuthology.orchestra.run.smithi170.stdout: "req_unlink_latency": { 2024-03-26T06:48:08.749 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.750 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.750 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.750 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.750 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_revoke_eviction": 0, 2024-03-26T06:48:08.750 INFO:teuthology.orchestra.run.smithi170.stdout: "cap_acquisition_throttle": 0, 2024-03-26T06:48:08.750 INFO:teuthology.orchestra.run.smithi170.stdout: "req_getvxattr_latency": { 2024-03-26T06:48:08.750 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.751 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.751 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.751 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:08.751 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.751 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_sessions": { 2024-03-26T06:48:08.751 INFO:teuthology.orchestra.run.smithi170.stdout: "session_count": 2, 2024-03-26T06:48:08.751 INFO:teuthology.orchestra.run.smithi170.stdout: "session_add": 3, 2024-03-26T06:48:08.751 INFO:teuthology.orchestra.run.smithi170.stdout: "session_remove": 1, 2024-03-26T06:48:08.752 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_open": 2, 2024-03-26T06:48:08.752 INFO:teuthology.orchestra.run.smithi170.stdout: "sessions_stale": 0, 2024-03-26T06:48:08.752 INFO:teuthology.orchestra.run.smithi170.stdout: "total_load": 4, 2024-03-26T06:48:08.752 INFO:teuthology.orchestra.run.smithi170.stdout: "average_load": 1, 2024-03-26T06:48:08.752 INFO:teuthology.orchestra.run.smithi170.stdout: "avg_session_uptime": 620, 2024-03-26T06:48:08.752 INFO:teuthology.orchestra.run.smithi170.stdout: "mdthresh_evicted": 0 2024-03-26T06:48:08.752 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.752 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-03-26T06:48:08.752 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-03-26T06:48:08.753 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-03-26T06:48:08.753 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:48:08.753 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:48:08.753 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:48:08.753 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:48:08.753 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:48:08.753 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:48:08.753 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:48:08.754 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:48:08.754 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:48:08.754 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:48:08.754 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:48:08.754 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:48:08.754 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:48:08.754 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-03-26T06:48:08.755 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:48:08.755 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-03-26T06:48:08.755 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:48:08.755 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:48:08.755 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:48:08.755 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:48:08.755 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:48:08.755 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:48:08.755 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:48:08.756 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-03-26T06:48:08.756 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:48:08.756 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:48:08.756 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:48:08.756 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-03-26T06:48:08.756 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-03-26T06:48:08.756 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-03-26T06:48:08.756 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:48:08.757 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:48:08.757 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:48:08.757 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:48:08.757 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 266805, 2024-03-26T06:48:08.757 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 121, 2024-03-26T06:48:08.757 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 0, 2024-03-26T06:48:08.757 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 0, 2024-03-26T06:48:08.757 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-03-26T06:48:08.758 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-03-26T06:48:08.758 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:48:08.758 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-03-26T06:48:08.758 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:48:08.758 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-03-26T06:48:08.758 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 8472, 2024-03-26T06:48:08.758 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 193, 2024-03-26T06:48:08.759 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:48:08.759 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:48:08.759 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-03-26T06:48:08.759 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-03-26T06:48:08.759 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 121307, 2024-03-26T06:48:08.759 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 948, 2024-03-26T06:48:08.759 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-03-26T06:48:08.759 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-03-26T06:48:08.759 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-03-26T06:48:08.760 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-03-26T06:48:08.760 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.760 INFO:teuthology.orchestra.run.smithi170.stdout: "objecter": { 2024-03-26T06:48:08.760 INFO:teuthology.orchestra.run.smithi170.stdout: "op_active": 0, 2024-03-26T06:48:08.760 INFO:teuthology.orchestra.run.smithi170.stdout: "op_laggy": 0, 2024-03-26T06:48:08.760 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send": 53, 2024-03-26T06:48:08.760 INFO:teuthology.orchestra.run.smithi170.stdout: "op_send_bytes": 42353, 2024-03-26T06:48:08.760 INFO:teuthology.orchestra.run.smithi170.stdout: "op_resend": 33, 2024-03-26T06:48:08.761 INFO:teuthology.orchestra.run.smithi170.stdout: "op_reply": 53, 2024-03-26T06:48:08.761 INFO:teuthology.orchestra.run.smithi170.stdout: "op_latency": { 2024-03-26T06:48:08.761 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 53, 2024-03-26T06:48:08.761 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.242994275, 2024-03-26T06:48:08.761 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.004584797 2024-03-26T06:48:08.761 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.761 INFO:teuthology.orchestra.run.smithi170.stdout: "op_inflight": 0, 2024-03-26T06:48:08.761 INFO:teuthology.orchestra.run.smithi170.stdout: "oplen_avg": { 2024-03-26T06:48:08.762 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 53, 2024-03-26T06:48:08.762 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 80 2024-03-26T06:48:08.762 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.762 INFO:teuthology.orchestra.run.smithi170.stdout: "op": 53, 2024-03-26T06:48:08.762 INFO:teuthology.orchestra.run.smithi170.stdout: "op_r": 0, 2024-03-26T06:48:08.762 INFO:teuthology.orchestra.run.smithi170.stdout: "op_w": 53, 2024-03-26T06:48:08.762 INFO:teuthology.orchestra.run.smithi170.stdout: "op_rmw": 0, 2024-03-26T06:48:08.763 INFO:teuthology.orchestra.run.smithi170.stdout: "op_pg": 0, 2024-03-26T06:48:08.763 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_stat": 0, 2024-03-26T06:48:08.763 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_create": 10, 2024-03-26T06:48:08.763 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_read": 0, 2024-03-26T06:48:08.763 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_write": 10, 2024-03-26T06:48:08.763 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writefull": 9, 2024-03-26T06:48:08.764 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_writesame": 0, 2024-03-26T06:48:08.764 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_append": 0, 2024-03-26T06:48:08.764 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_zero": 0, 2024-03-26T06:48:08.764 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_truncate": 0, 2024-03-26T06:48:08.764 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_delete": 6, 2024-03-26T06:48:08.765 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_mapext": 0, 2024-03-26T06:48:08.765 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_sparse_read": 0, 2024-03-26T06:48:08.765 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_clonerange": 0, 2024-03-26T06:48:08.765 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_getxattr": 0, 2024-03-26T06:48:08.765 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_setxattr": 20, 2024-03-26T06:48:08.765 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_cmpxattr": 0, 2024-03-26T06:48:08.765 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_rmxattr": 0, 2024-03-26T06:48:08.765 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_resetxattrs": 0, 2024-03-26T06:48:08.766 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_call": 0, 2024-03-26T06:48:08.766 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_watch": 0, 2024-03-26T06:48:08.766 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_notify": 0, 2024-03-26T06:48:08.766 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_src_cmpxattr": 0, 2024-03-26T06:48:08.767 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls": 0, 2024-03-26T06:48:08.767 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_pgls_filter": 0, 2024-03-26T06:48:08.767 INFO:teuthology.orchestra.run.smithi170.stdout: "osdop_other": 0, 2024-03-26T06:48:08.768 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_active": 0, 2024-03-26T06:48:08.768 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_send": 0, 2024-03-26T06:48:08.768 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_resend": 0, 2024-03-26T06:48:08.768 INFO:teuthology.orchestra.run.smithi170.stdout: "linger_ping": 0, 2024-03-26T06:48:08.769 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_active": 0, 2024-03-26T06:48:08.769 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_send": 0, 2024-03-26T06:48:08.769 INFO:teuthology.orchestra.run.smithi170.stdout: "poolop_resend": 0, 2024-03-26T06:48:08.769 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_active": 0, 2024-03-26T06:48:08.769 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_send": 0, 2024-03-26T06:48:08.769 INFO:teuthology.orchestra.run.smithi170.stdout: "poolstat_resend": 0, 2024-03-26T06:48:08.770 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_active": 0, 2024-03-26T06:48:08.770 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_send": 0, 2024-03-26T06:48:08.770 INFO:teuthology.orchestra.run.smithi170.stdout: "statfs_resend": 0, 2024-03-26T06:48:08.770 INFO:teuthology.orchestra.run.smithi170.stdout: "command_active": 0, 2024-03-26T06:48:08.770 INFO:teuthology.orchestra.run.smithi170.stdout: "command_send": 0, 2024-03-26T06:48:08.770 INFO:teuthology.orchestra.run.smithi170.stdout: "command_resend": 0, 2024-03-26T06:48:08.770 INFO:teuthology.orchestra.run.smithi170.stdout: "map_epoch": 99, 2024-03-26T06:48:08.771 INFO:teuthology.orchestra.run.smithi170.stdout: "map_full": 0, 2024-03-26T06:48:08.771 INFO:teuthology.orchestra.run.smithi170.stdout: "map_inc": 13, 2024-03-26T06:48:08.771 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_sessions": 9, 2024-03-26T06:48:08.771 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_open": 9, 2024-03-26T06:48:08.771 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_session_close": 0, 2024-03-26T06:48:08.771 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_laggy": 0, 2024-03-26T06:48:08.771 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_wr": 23, 2024-03-26T06:48:08.771 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_rd": 0, 2024-03-26T06:48:08.772 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_del": 2 2024-03-26T06:48:08.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.772 INFO:teuthology.orchestra.run.smithi170.stdout: "oft": { 2024-03-26T06:48:08.772 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_objs": 1, 2024-03-26T06:48:08.772 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_kv_pairs": 0, 2024-03-26T06:48:08.772 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_updates": 0, 2024-03-26T06:48:08.772 INFO:teuthology.orchestra.run.smithi170.stdout: "omap_total_removes": 0 2024-03-26T06:48:08.772 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: "osd-slow-ops": { 2024-03-26T06:48:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: "slow_ops_count": 0 2024-03-26T06:48:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: "purge_queue": { 2024-03-26T06:48:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops": 0, 2024-03-26T06:48:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_ops_high_water": 0, 2024-03-26T06:48:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing": 0, 2024-03-26T06:48:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executing_high_water": 0, 2024-03-26T06:48:08.773 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed_ops": 0, 2024-03-26T06:48:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_executed": 0, 2024-03-26T06:48:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: "pq_item_in_journal": 0 2024-03-26T06:48:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:48:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-26T06:48:08.774 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 1294, 2024-03-26T06:48:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 1813315, 2024-03-26T06:48:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 1294, 2024-03-26T06:48:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-26T06:48:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-26T06:48:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 1294, 2024-03-26T06:48:08.775 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 1813315, 2024-03-26T06:48:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_bytes": { 2024-03-26T06:48:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:08.776 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-26T06:48:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-26T06:48:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-26T06:48:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-26T06:48:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 53, 2024-03-26T06:48:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 42353, 2024-03-26T06:48:08.777 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 53, 2024-03-26T06:48:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 42353, 2024-03-26T06:48:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.778 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-objecter_ops": { 2024-03-26T06:48:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 1024, 2024-03-26T06:48:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-26T06:48:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-26T06:48:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-26T06:48:08.779 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 53, 2024-03-26T06:48:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 53, 2024-03-26T06:48:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 53, 2024-03-26T06:48:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 53, 2024-03-26T06:48:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.780 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle": { 2024-03-26T06:48:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-26T06:48:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 0, 2024-03-26T06:48:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 0, 2024-03-26T06:48:08.781 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 0, 2024-03-26T06:48:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-26T06:48:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-26T06:48:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 0, 2024-03-26T06:48:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 0, 2024-03-26T06:48:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.782 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-write_buf_throttle-0x55ff6c7506e0": { 2024-03-26T06:48:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 3758096384, 2024-03-26T06:48:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:08.783 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 14, 2024-03-26T06:48:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 30589, 2024-03-26T06:48:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 14, 2024-03-26T06:48:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-26T06:48:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-26T06:48:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 10, 2024-03-26T06:48:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 30589, 2024-03-26T06:48:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:08.784 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:08.785 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:08.785 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-26T06:48:08.785 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:08 smithi170 ceph-mon[36912]: pgmap v329: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-03-26T06:48:08.786 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:08 smithi170 ceph-mon[36912]: osdmap e99: 12 total, 12 up, 12 in 2024-03-26T06:48:08.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:08 smithi038 ceph-mon[32265]: pgmap v329: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-03-26T06:48:08.835 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:08 smithi038 ceph-mon[32265]: osdmap e99: 12 total, 12 up, 12 in 2024-03-26T06:48:08.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:08 smithi121 ceph-mon[36349]: pgmap v329: 97 pgs: 97 active+clean; 13 GiB data, 63 GiB used, 1010 GiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-03-26T06:48:08.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:08 smithi121 ceph-mon[36349]: osdmap e99: 12 total, 12 up, 12 in 2024-03-26T06:48:09.481 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-26T06:48:09.482 INFO:tasks.check_counter:Daemon mds.i mds.imported=2 2024-03-26T06:48:09.482 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=5 2024-03-26T06:48:09.482 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-26T06:48:09.482 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-26T06:48:09.482 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-26T06:48:09.483 DEBUG:tasks.check_counter:Getting stats from l 2024-03-26T06:48:09.483 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f shell --fsid c2824556-eb39-11ee-95d0-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-26T06:48:09.723 INFO:teuthology.orchestra.run.smithi170.stderr:Inferring config /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/mon.c/config 2024-03-26T06:48:10.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:10 smithi038 ceph-mon[32265]: pgmap v331: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-26T06:48:10.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:10 smithi121 ceph-mon[36349]: pgmap v331: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-26T06:48:10.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:10 smithi170 ceph-mon[36912]: pgmap v331: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-26T06:48:11.539 INFO:teuthology.orchestra.run.smithi170.stdout:{ 2024-03-26T06:48:11.540 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-0": { 2024-03-26T06:48:11.540 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 0, 2024-03-26T06:48:11.540 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 2, 2024-03-26T06:48:11.540 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 0, 2024-03-26T06:48:11.540 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 0, 2024-03-26T06:48:11.540 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 4, 2024-03-26T06:48:11.540 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 0, 2024-03-26T06:48:11.540 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.001073763, 2024-03-26T06:48:11.541 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.000000000, 2024-03-26T06:48:11.541 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.000274761, 2024-03-26T06:48:11.541 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:48:11.541 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:11.541 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:11.541 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:11.541 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:11.541 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:11.541 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:11.542 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:11.542 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:11.542 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:11.542 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:11.542 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-26T06:48:11.542 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 0 2024-03-26T06:48:11.542 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:11.542 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-1": { 2024-03-26T06:48:11.543 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 6, 2024-03-26T06:48:11.543 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 166, 2024-03-26T06:48:11.543 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 3344, 2024-03-26T06:48:11.543 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 184400, 2024-03-26T06:48:11.543 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 5, 2024-03-26T06:48:11.543 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 3, 2024-03-26T06:48:11.543 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.011470590, 2024-03-26T06:48:11.543 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.007455736, 2024-03-26T06:48:11.544 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.001767376, 2024-03-26T06:48:11.544 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-26T06:48:11.544 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:11.544 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 164, 2024-03-26T06:48:11.544 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.016642498, 2024-03-26T06:48:11.544 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000101478 2024-03-26T06:48:11.544 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:11.544 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:11.545 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:11.545 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:11.545 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:11.545 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:11.545 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-26T06:48:11.545 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 184400 2024-03-26T06:48:11.545 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:11.545 INFO:teuthology.orchestra.run.smithi170.stdout: "AsyncMessenger::Worker-2": { 2024-03-26T06:48:11.545 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_messages": 227, 2024-03-26T06:48:11.546 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages": 216, 2024-03-26T06:48:11.546 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_bytes": 1576272, 2024-03-26T06:48:11.546 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_bytes": 129312, 2024-03-26T06:48:11.546 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_created_connections": 2, 2024-03-26T06:48:11.546 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_active_connections": 1, 2024-03-26T06:48:11.546 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_total_time": 0.044730734, 2024-03-26T06:48:11.546 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_send_time": 0.013815793, 2024-03-26T06:48:11.546 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_recv_time": 0.034853302, 2024-03-26T06:48:11.547 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_running_fast_dispatch_time": 0.005495771, 2024-03-26T06:48:11.547 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_messages_queue_lat": { 2024-03-26T06:48:11.547 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 215, 2024-03-26T06:48:11.547 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.012166566, 2024-03-26T06:48:11.547 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000056588 2024-03-26T06:48:11.547 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:11.547 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_handle_ack_lat": { 2024-03-26T06:48:11.547 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:11.547 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:11.550 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:11.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:11.551 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_recv_encrypted_bytes": 1576272, 2024-03-26T06:48:11.551 INFO:teuthology.orchestra.run.smithi170.stdout: "msgr_send_encrypted_bytes": 129312 2024-03-26T06:48:11.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:11.551 INFO:teuthology.orchestra.run.smithi170.stdout: "cct": { 2024-03-26T06:48:11.551 INFO:teuthology.orchestra.run.smithi170.stdout: "total_workers": 0, 2024-03-26T06:48:11.551 INFO:teuthology.orchestra.run.smithi170.stdout: "unhealthy_workers": 0 2024-03-26T06:48:11.551 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:11.552 INFO:teuthology.orchestra.run.smithi170.stdout: "mempool": { 2024-03-26T06:48:11.552 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_bytes": 0, 2024-03-26T06:48:11.552 INFO:teuthology.orchestra.run.smithi170.stdout: "bloom_filter_items": 0, 2024-03-26T06:48:11.552 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_bytes": 0, 2024-03-26T06:48:11.552 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_alloc_items": 0, 2024-03-26T06:48:11.552 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_bytes": 0, 2024-03-26T06:48:11.552 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_data_items": 0, 2024-03-26T06:48:11.552 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-26T06:48:11.553 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_onode_items": 0, 2024-03-26T06:48:11.553 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-26T06:48:11.553 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_meta_items": 0, 2024-03-26T06:48:11.553 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_bytes": 0, 2024-03-26T06:48:11.553 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_other_items": 0, 2024-03-26T06:48:11.553 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-26T06:48:11.553 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_cache_buffer_items": 0, 2024-03-26T06:48:11.553 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_bytes": 0, 2024-03-26T06:48:11.554 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_extent_items": 0, 2024-03-26T06:48:11.554 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_bytes": 0, 2024-03-26T06:48:11.554 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_blob_items": 0, 2024-03-26T06:48:11.554 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-26T06:48:11.554 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_shared_blob_items": 0, 2024-03-26T06:48:11.554 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-26T06:48:11.554 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_inline_bl_items": 0, 2024-03-26T06:48:11.554 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_bytes": 0, 2024-03-26T06:48:11.554 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_fsck_items": 0, 2024-03-26T06:48:11.555 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_bytes": 0, 2024-03-26T06:48:11.555 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_txc_items": 0, 2024-03-26T06:48:11.555 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-26T06:48:11.555 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_deferred_items": 0, 2024-03-26T06:48:11.555 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_bytes": 0, 2024-03-26T06:48:11.555 INFO:teuthology.orchestra.run.smithi170.stdout: "bluestore_writing_items": 0, 2024-03-26T06:48:11.555 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_bytes": 0, 2024-03-26T06:48:11.555 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_items": 0, 2024-03-26T06:48:11.556 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_bytes": 0, 2024-03-26T06:48:11.556 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_reader_items": 0, 2024-03-26T06:48:11.556 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_bytes": 0, 2024-03-26T06:48:11.556 INFO:teuthology.orchestra.run.smithi170.stdout: "bluefs_file_writer_items": 0, 2024-03-26T06:48:11.556 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_bytes": 148977, 2024-03-26T06:48:11.556 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_anon_items": 30, 2024-03-26T06:48:11.556 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_bytes": 0, 2024-03-26T06:48:11.556 INFO:teuthology.orchestra.run.smithi170.stdout: "buffer_meta_items": 0, 2024-03-26T06:48:11.557 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_bytes": 0, 2024-03-26T06:48:11.557 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_items": 0, 2024-03-26T06:48:11.557 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_bytes": 0, 2024-03-26T06:48:11.557 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_mapbl_items": 0, 2024-03-26T06:48:11.557 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_bytes": 0, 2024-03-26T06:48:11.557 INFO:teuthology.orchestra.run.smithi170.stdout: "osd_pglog_items": 0, 2024-03-26T06:48:11.557 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_bytes": 0, 2024-03-26T06:48:11.557 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_items": 0, 2024-03-26T06:48:11.557 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_bytes": 0, 2024-03-26T06:48:11.558 INFO:teuthology.orchestra.run.smithi170.stdout: "osdmap_mapping_items": 0, 2024-03-26T06:48:11.558 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_bytes": 0, 2024-03-26T06:48:11.558 INFO:teuthology.orchestra.run.smithi170.stdout: "pgmap_items": 0, 2024-03-26T06:48:11.558 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_bytes": 928, 2024-03-26T06:48:11.558 INFO:teuthology.orchestra.run.smithi170.stdout: "mds_co_items": 2, 2024-03-26T06:48:11.558 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_bytes": 0, 2024-03-26T06:48:11.558 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_1_items": 0, 2024-03-26T06:48:11.558 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_bytes": 0, 2024-03-26T06:48:11.559 INFO:teuthology.orchestra.run.smithi170.stdout: "unittest_2_items": 0 2024-03-26T06:48:11.559 INFO:teuthology.orchestra.run.smithi170.stdout: }, 2024-03-26T06:48:11.559 INFO:teuthology.orchestra.run.smithi170.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-26T06:48:11.559 INFO:teuthology.orchestra.run.smithi170.stdout: "val": 0, 2024-03-26T06:48:11.559 INFO:teuthology.orchestra.run.smithi170.stdout: "max": 104857600, 2024-03-26T06:48:11.559 INFO:teuthology.orchestra.run.smithi170.stdout: "get_started": 0, 2024-03-26T06:48:11.559 INFO:teuthology.orchestra.run.smithi170.stdout: "get": 229, 2024-03-26T06:48:11.559 INFO:teuthology.orchestra.run.smithi170.stdout: "get_sum": 1544292, 2024-03-26T06:48:11.559 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_fail": 0, 2024-03-26T06:48:11.560 INFO:teuthology.orchestra.run.smithi170.stdout: "get_or_fail_success": 229, 2024-03-26T06:48:11.560 INFO:teuthology.orchestra.run.smithi170.stdout: "take": 0, 2024-03-26T06:48:11.560 INFO:teuthology.orchestra.run.smithi170.stdout: "take_sum": 0, 2024-03-26T06:48:11.560 INFO:teuthology.orchestra.run.smithi170.stdout: "put": 229, 2024-03-26T06:48:11.560 INFO:teuthology.orchestra.run.smithi170.stdout: "put_sum": 1544292, 2024-03-26T06:48:11.560 INFO:teuthology.orchestra.run.smithi170.stdout: "wait": { 2024-03-26T06:48:11.560 INFO:teuthology.orchestra.run.smithi170.stdout: "avgcount": 0, 2024-03-26T06:48:11.560 INFO:teuthology.orchestra.run.smithi170.stdout: "sum": 0.000000000, 2024-03-26T06:48:11.561 INFO:teuthology.orchestra.run.smithi170.stdout: "avgtime": 0.000000000 2024-03-26T06:48:11.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:11.561 INFO:teuthology.orchestra.run.smithi170.stdout: } 2024-03-26T06:48:11.561 INFO:teuthology.orchestra.run.smithi170.stdout:} 2024-03-26T06:48:12.222 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-26T06:48:12.222 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-26T06:48:12.224 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-26T06:48:12.224 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-26T06:48:12.224 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-26T06:48:12.224 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-03-26T06:48:12.225 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-26T06:48:12.238 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-26T06:48:12.238 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:48:12.238 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-26T06:48:12.260 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2024-03-26T06:48:12.260 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-26T06:48:12.318 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:48:12.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.469+0000 7fcf3d461640 1 -- 172.21.15.38:0/3489068702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf38105530 msgr2=0x7fcf3810d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:12.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.469+0000 7fcf3d461640 1 --2- 172.21.15.38:0/3489068702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf38105530 0x7fcf3810d100 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fcf30009ca0 tx=0x7fcf3002c0f0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.469+0000 7fcf3d461640 1 -- 172.21.15.38:0/3489068702 shutdown_connections 2024-03-26T06:48:12.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.469+0000 7fcf3d461640 1 --2- 172.21.15.38:0/3489068702 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf38105530 0x7fcf3810d100 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.469+0000 7fcf3d461640 1 --2- 172.21.15.38:0/3489068702 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf38100e10 0x7fcf38104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.475 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.469+0000 7fcf3d461640 1 --2- 172.21.15.38:0/3489068702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf38100460 0x7fcf38100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.469+0000 7fcf3d461640 1 -- 172.21.15.38:0/3489068702 >> 172.21.15.38:0/3489068702 conn(0x7fcf380772f0 msgr2=0x7fcf38077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:12.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.469+0000 7fcf3d461640 1 -- 172.21.15.38:0/3489068702 shutdown_connections 2024-03-26T06:48:12.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.469+0000 7fcf3d461640 1 -- 172.21.15.38:0/3489068702 wait complete. 2024-03-26T06:48:12.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.469+0000 7fcf3d461640 1 Processor -- start 2024-03-26T06:48:12.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.469+0000 7fcf3d461640 1 -- start start 2024-03-26T06:48:12.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf3d461640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf38100460 0x7fcf3819d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:12.476 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf3d461640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf38105530 0x7fcf3819dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:12.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf3d461640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf381978d0 0x7fcf38197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:12.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf3d461640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf381105d0 con 0x7fcf38100460 2024-03-26T06:48:12.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf3d461640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38110450 con 0x7fcf381978d0 2024-03-26T06:48:12.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf3d461640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38110750 con 0x7fcf38105530 2024-03-26T06:48:12.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf36ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf38100460 0x7fcf3819d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:12.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf36ffd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf38100460 0x7fcf3819d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54218/0 (socket says 172.21.15.38:54218) 2024-03-26T06:48:12.477 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf36ffd640 1 -- 172.21.15.38:0/3678414198 learned_addr learned my addr 172.21.15.38:0/3678414198 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:12.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf2ffff640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf38105530 0x7fcf3819dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:12.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf377fe640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf381978d0 0x7fcf38197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:12.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf36ffd640 1 -- 172.21.15.38:0/3678414198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf38105530 msgr2=0x7fcf3819dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:12.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf36ffd640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf38105530 0x7fcf3819dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf36ffd640 1 -- 172.21.15.38:0/3678414198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf381978d0 msgr2=0x7fcf38197d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:12.478 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf36ffd640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf381978d0 0x7fcf38197d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf36ffd640 1 -- 172.21.15.38:0/3678414198 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf381983f0 con 0x7fcf38100460 2024-03-26T06:48:12.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf36ffd640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf38100460 0x7fcf3819d720 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800c920 tx=0x7fcf2800cd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:12.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf34ff9640 1 -- 172.21.15.38:0/3678414198 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf28007bb0 con 0x7fcf38100460 2024-03-26T06:48:12.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf34ff9640 1 -- 172.21.15.38:0/3678414198 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf28007d50 con 0x7fcf38100460 2024-03-26T06:48:12.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf3d461640 1 -- 172.21.15.38:0/3678414198 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf381986e0 con 0x7fcf38100460 2024-03-26T06:48:12.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.470+0000 7fcf34ff9640 1 -- 172.21.15.38:0/3678414198 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf280187d0 con 0x7fcf38100460 2024-03-26T06:48:12.479 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.471+0000 7fcf3d461640 1 -- 172.21.15.38:0/3678414198 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf381a4930 con 0x7fcf38100460 2024-03-26T06:48:12.480 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.472+0000 7fcf3d461640 1 -- 172.21.15.38:0/3678414198 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf380680e0 con 0x7fcf38100460 2024-03-26T06:48:12.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.473+0000 7fcf34ff9640 1 -- 172.21.15.38:0/3678414198 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcf2800f450 con 0x7fcf38100460 2024-03-26T06:48:12.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.473+0000 7fcf34ff9640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fcf240a2970 0x7fcf240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:12.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.474+0000 7fcf2ffff640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fcf240a2970 0x7fcf240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:12.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.474+0000 7fcf34ff9640 1 -- 172.21.15.38:0/3678414198 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8629+0+0 (secure 0 0 0) 0x7fcf280cef90 con 0x7fcf38100460 2024-03-26T06:48:12.481 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.474+0000 7fcf2ffff640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fcf240a2970 0x7fcf240a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcf38104e30 tx=0x7fcf2000a2d0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:12.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.478+0000 7fcf34ff9640 1 -- 172.21.15.38:0/3678414198 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcf28014030 con 0x7fcf38100460 2024-03-26T06:48:12.666 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.663+0000 7fcf3d461640 1 -- 172.21.15.38:0/3678414198 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf38109830 con 0x7fcf38100460 2024-03-26T06:48:12.667 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.664+0000 7fcf34ff9640 1 -- 172.21.15.38:0/3678414198 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+23231 (secure 0 0 0) 0x7fcf2809bc80 con 0x7fcf38100460 2024-03-26T06:48:12.668 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:48:12.668 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":99,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:48:07.543242+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":99,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/2716021457":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/2750459261":"2024-03-27T06:37:32.618701+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/1675065388":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/4027757314":"2024-03-27T06:37:32.618701+0000","172.21.15.38:6800/2716021457":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/3320689790":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/3570258347":"2024-03-27T06:37:32.618701+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:48:12.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 -- 172.21.15.38:0/3678414198 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fcf240a2970 msgr2=0x7fcf240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:12.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fcf240a2970 0x7fcf240a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fcf38104e30 tx=0x7fcf2000a2d0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 -- 172.21.15.38:0/3678414198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf38100460 msgr2=0x7fcf3819d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:12.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf38100460 0x7fcf3819d720 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fcf2800c920 tx=0x7fcf2800cd20 comp rx=0 tx=0).stop 2024-03-26T06:48:12.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 -- 172.21.15.38:0/3678414198 shutdown_connections 2024-03-26T06:48:12.670 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fcf240a2970 0x7fcf240a4e30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf381978d0 0x7fcf38197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fcf38105530 0x7fcf3819dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 --2- 172.21.15.38:0/3678414198 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcf38100460 0x7fcf3819d720 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.671 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 -- 172.21.15.38:0/3678414198 >> 172.21.15.38:0/3678414198 conn(0x7fcf380772f0 msgr2=0x7fcf380685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:12.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 -- 172.21.15.38:0/3678414198 shutdown_connections 2024-03-26T06:48:12.672 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.666+0000 7fcf3d461640 1 -- 172.21.15.38:0/3678414198 wait complete. 2024-03-26T06:48:12.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:12 smithi038 ceph-mon[32265]: pgmap v332: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-26T06:48:12.681 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T06:30:52.485897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-26T06:35:24.501259+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-26T06:35:25.675299+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '99', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 99, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T06:48:12.681 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-26T06:48:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.783+0000 7f6f1d11b640 1 -- 172.21.15.38:0/4129610346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f18100480 msgr2=0x7f6f18100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.783+0000 7f6f1d11b640 1 --2- 172.21.15.38:0/4129610346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f18100480 0x7f6f18100860 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6f08009900 tx=0x7f6f0802bc30 comp rx=0 tx=0).stop 2024-03-26T06:48:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.783+0000 7f6f1d11b640 1 -- 172.21.15.38:0/4129610346 shutdown_connections 2024-03-26T06:48:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.783+0000 7f6f1d11b640 1 --2- 172.21.15.38:0/4129610346 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f18105550 0x7f6f1810d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.783+0000 7f6f1d11b640 1 --2- 172.21.15.38:0/4129610346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f18100e30 0x7f6f18104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.783+0000 7f6f1d11b640 1 --2- 172.21.15.38:0/4129610346 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f18100480 0x7f6f18100860 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.783+0000 7f6f1d11b640 1 -- 172.21.15.38:0/4129610346 >> 172.21.15.38:0/4129610346 conn(0x7f6f18077350 msgr2=0x7f6f18077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.783+0000 7f6f1d11b640 1 -- 172.21.15.38:0/4129610346 shutdown_connections 2024-03-26T06:48:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.783+0000 7f6f1d11b640 1 -- 172.21.15.38:0/4129610346 wait complete. 2024-03-26T06:48:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f1d11b640 1 Processor -- start 2024-03-26T06:48:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f1d11b640 1 -- start start 2024-03-26T06:48:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f1d11b640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f18100480 0x7f6f18198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f1d11b640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f18105550 0x7f6f18198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f1d11b640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f18199580 0x7f6f181a3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f1d11b640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f181106b0 con 0x7f6f18100480 2024-03-26T06:48:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f1d11b640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f18110530 con 0x7f6f18199580 2024-03-26T06:48:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f1d11b640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f18110830 con 0x7f6f18105550 2024-03-26T06:48:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f16d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f18100480 0x7f6f18198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f16d9d640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f18100480 0x7f6f18198ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54226/0 (socket says 172.21.15.38:54226) 2024-03-26T06:48:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f16d9d640 1 -- 172.21.15.38:0/1103009775 learned_addr learned my addr 172.21.15.38:0/1103009775 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f1759e640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f18199580 0x7f6f181a3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f16d9d640 1 -- 172.21.15.38:0/1103009775 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f18105550 msgr2=0x7f6f18198ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f16d9d640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f18105550 0x7f6f18198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f16d9d640 1 -- 172.21.15.38:0/1103009775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f18199580 msgr2=0x7f6f181a3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f16d9d640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f18199580 0x7f6f181a3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f16d9d640 1 -- 172.21.15.38:0/1103009775 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f181a4300 con 0x7f6f18100480 2024-03-26T06:48:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.784+0000 7f6f1759e640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f18199580 0x7f6f181a3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:48:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.785+0000 7f6f16d9d640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f18100480 0x7f6f18198ab0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6f0802c240 tx=0x7f6f080042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.785+0000 7f6f14d99640 1 -- 172.21.15.38:0/1103009775 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08043020 con 0x7f6f18100480 2024-03-26T06:48:12.792 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.785+0000 7f6f14d99640 1 -- 172.21.15.38:0/1103009775 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f080054e0 con 0x7f6f18100480 2024-03-26T06:48:12.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.785+0000 7f6f1d11b640 1 -- 172.21.15.38:0/1103009775 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f181a4590 con 0x7f6f18100480 2024-03-26T06:48:12.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.785+0000 7f6f14d99640 1 -- 172.21.15.38:0/1103009775 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f08003b50 con 0x7f6f18100480 2024-03-26T06:48:12.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.785+0000 7f6f1d11b640 1 -- 172.21.15.38:0/1103009775 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f181a4ae0 con 0x7f6f18100480 2024-03-26T06:48:12.793 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.786+0000 7f6f1d11b640 1 -- 172.21.15.38:0/1103009775 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f18068530 con 0x7f6f18100480 2024-03-26T06:48:12.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.787+0000 7f6f14d99640 1 -- 172.21.15.38:0/1103009775 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f6f08046020 con 0x7f6f18100480 2024-03-26T06:48:12.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.787+0000 7f6f14d99640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f6f040a2970 0x7f6f040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:12.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.787+0000 7f6f0ffff640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f6f040a2970 0x7f6f040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:12.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.787+0000 7f6f14d99640 1 -- 172.21.15.38:0/1103009775 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8629+0+0 (secure 0 0 0) 0x7f6f080f0fe0 con 0x7f6f18100480 2024-03-26T06:48:12.794 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.788+0000 7f6f0ffff640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f6f040a2970 0x7f6f040a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6f000058c0 tx=0x7f6f00005850 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:12.795 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.792+0000 7f6f14d99640 1 -- 172.21.15.38:0/1103009775 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6f0804d020 con 0x7f6f18100480 2024-03-26T06:48:12.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:12 smithi121 ceph-mon[36349]: pgmap v332: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-26T06:48:12.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:12 smithi170 ceph-mon[36912]: pgmap v332: 97 pgs: 97 active+clean; 13 GiB data, 62 GiB used, 1011 GiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-26T06:48:12.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.973+0000 7f6f1d11b640 1 -- 172.21.15.38:0/1103009775 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f18108780 con 0x7f6f18100480 2024-03-26T06:48:12.977 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.974+0000 7f6f14d99640 1 -- 172.21.15.38:0/1103009775 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f080bdd80 con 0x7f6f18100480 2024-03-26T06:48:12.978 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2024-03-26T06:48:12.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 -- 172.21.15.38:0/1103009775 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f6f040a2970 msgr2=0x7f6f040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:12.980 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f6f040a2970 0x7f6f040a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6f000058c0 tx=0x7f6f00005850 comp rx=0 tx=0).stop 2024-03-26T06:48:12.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 -- 172.21.15.38:0/1103009775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f18100480 msgr2=0x7f6f18198ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:12.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f18100480 0x7f6f18198ab0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6f0802c240 tx=0x7f6f080042f0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.981 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 -- 172.21.15.38:0/1103009775 shutdown_connections 2024-03-26T06:48:12.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f6f040a2970 0x7f6f040a4e30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6f18199580 0x7f6f181a3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f6f18105550 0x7f6f18198ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.982 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 --2- 172.21.15.38:0/1103009775 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f18100480 0x7f6f18198ab0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:12.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 -- 172.21.15.38:0/1103009775 >> 172.21.15.38:0/1103009775 conn(0x7f6f18077350 msgr2=0x7f6f180fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:12.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 -- 172.21.15.38:0/1103009775 shutdown_connections 2024-03-26T06:48:12.983 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:12.976+0000 7f6f1d11b640 1 -- 172.21.15.38:0/1103009775 wait complete. 2024-03-26T06:48:12.991 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-26T06:48:13.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.093+0000 7ff4d2854640 1 -- 172.21.15.38:0/494559099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc1065c0 msgr2=0x7ff4cc10e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.093+0000 7ff4d2854640 1 --2- 172.21.15.38:0/494559099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc1065c0 0x7ff4cc10e190 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff4c4009ca0 tx=0x7ff4c402c0f0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.093+0000 7ff4d2854640 1 -- 172.21.15.38:0/494559099 shutdown_connections 2024-03-26T06:48:13.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.093+0000 7ff4d2854640 1 --2- 172.21.15.38:0/494559099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc1065c0 0x7ff4cc10e190 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.093+0000 7ff4d2854640 1 --2- 172.21.15.38:0/494559099 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4cc101ea0 0x7ff4cc105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.093+0000 7ff4d2854640 1 --2- 172.21.15.38:0/494559099 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4cc1014f0 0x7ff4cc1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.093+0000 7ff4d2854640 1 -- 172.21.15.38:0/494559099 >> 172.21.15.38:0/494559099 conn(0x7ff4cc0786f0 msgr2=0x7ff4cc078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:13.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.093+0000 7ff4d2854640 1 -- 172.21.15.38:0/494559099 shutdown_connections 2024-03-26T06:48:13.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.093+0000 7ff4d2854640 1 -- 172.21.15.38:0/494559099 wait complete. 2024-03-26T06:48:13.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.094+0000 7ff4d2854640 1 Processor -- start 2024-03-26T06:48:13.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.094+0000 7ff4d2854640 1 -- start start 2024-03-26T06:48:13.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.094+0000 7ff4d2854640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4cc1014f0 0x7ff4cc19d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.094+0000 7ff4d2854640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc1065c0 0x7ff4cc19dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.094+0000 7ff4d2854640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4cc197920 0x7ff4cc197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.094+0000 7ff4d2854640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc111680 con 0x7ff4cc1065c0 2024-03-26T06:48:13.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.094+0000 7ff4d2854640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc111500 con 0x7ff4cc1014f0 2024-03-26T06:48:13.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.094+0000 7ff4d2854640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc111800 con 0x7ff4cc197920 2024-03-26T06:48:13.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.094+0000 7ff4cb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc1065c0 0x7ff4cc19dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:13.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4cbfff640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4cc1014f0 0x7ff4cc19d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:13.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.094+0000 7ff4cb7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc1065c0 0x7ff4cc19dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54250/0 (socket says 172.21.15.38:54250) 2024-03-26T06:48:13.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4cb7fe640 1 -- 172.21.15.38:0/3363719011 learned_addr learned my addr 172.21.15.38:0/3363719011 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:13.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4d0df1640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4cc197920 0x7ff4cc197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:13.102 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4cb7fe640 1 -- 172.21.15.38:0/3363719011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4cc197920 msgr2=0x7ff4cc197dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4cb7fe640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4cc197920 0x7ff4cc197dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4cb7fe640 1 -- 172.21.15.38:0/3363719011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4cc1014f0 msgr2=0x7ff4cc19d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4cb7fe640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4cc1014f0 0x7ff4cc19d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4cb7fe640 1 -- 172.21.15.38:0/3363719011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc198440 con 0x7ff4cc1065c0 2024-03-26T06:48:13.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4d0df1640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4cc197920 0x7ff4cc197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:48:13.103 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4cbfff640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4cc1014f0 0x7ff4cc19d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:48:13.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4cb7fe640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc1065c0 0x7ff4cc19dcb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0005fd0 tx=0x7ff4c000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:13.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4c97fa640 1 -- 172.21.15.38:0/3363719011 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0017070 con 0x7ff4cc1065c0 2024-03-26T06:48:13.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4c97fa640 1 -- 172.21.15.38:0/3363719011 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4c0009c30 con 0x7ff4cc1065c0 2024-03-26T06:48:13.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4d2854640 1 -- 172.21.15.38:0/3363719011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc198730 con 0x7ff4cc1065c0 2024-03-26T06:48:13.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4c97fa640 1 -- 172.21.15.38:0/3363719011 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0010460 con 0x7ff4cc1065c0 2024-03-26T06:48:13.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.095+0000 7ff4d2854640 1 -- 172.21.15.38:0/3363719011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4cc1a4980 con 0x7ff4cc1065c0 2024-03-26T06:48:13.104 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.096+0000 7ff4d2854640 1 -- 172.21.15.38:0/3363719011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4cc068530 con 0x7ff4cc1065c0 2024-03-26T06:48:13.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.097+0000 7ff4c97fa640 1 -- 172.21.15.38:0/3363719011 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff4c0003e70 con 0x7ff4cc1065c0 2024-03-26T06:48:13.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.098+0000 7ff4c97fa640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff4b40a2970 0x7ff4b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.098+0000 7ff4cbfff640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff4b40a2970 0x7ff4b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:13.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.098+0000 7ff4c97fa640 1 -- 172.21.15.38:0/3363719011 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8629+0+0 (secure 0 0 0) 0x7ff4c00cf2f0 con 0x7ff4cc1065c0 2024-03-26T06:48:13.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.098+0000 7ff4cbfff640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff4b40a2970 0x7ff4b40a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc198a60 tx=0x7ff4bc00b390 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:13.106 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.102+0000 7ff4c97fa640 1 -- 172.21.15.38:0/3363719011 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff4c0020030 con 0x7ff4cc1065c0 2024-03-26T06:48:13.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.282+0000 7ff4d2854640 1 -- 172.21.15.38:0/3363719011 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff4cc101ea0 con 0x7ff4cc1065c0 2024-03-26T06:48:13.286 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.283+0000 7ff4c97fa640 1 -- 172.21.15.38:0/3363719011 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff4c009bf80 con 0x7ff4cc1065c0 2024-03-26T06:48:13.287 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2024-03-26T06:48:13.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 -- 172.21.15.38:0/3363719011 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff4b40a2970 msgr2=0x7ff4b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.288 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff4b40a2970 0x7ff4b40a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff4cc198a60 tx=0x7ff4bc00b390 comp rx=0 tx=0).stop 2024-03-26T06:48:13.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 -- 172.21.15.38:0/3363719011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc1065c0 msgr2=0x7ff4cc19dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc1065c0 0x7ff4cc19dcb0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0005fd0 tx=0x7ff4c000dde0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 -- 172.21.15.38:0/3363719011 shutdown_connections 2024-03-26T06:48:13.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7ff4b40a2970 0x7ff4b40a4e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7ff4cc197920 0x7ff4cc197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.289 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4cc1065c0 0x7ff4cc19dcb0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 --2- 172.21.15.38:0/3363719011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4cc1014f0 0x7ff4cc19d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 -- 172.21.15.38:0/3363719011 >> 172.21.15.38:0/3363719011 conn(0x7ff4cc0786f0 msgr2=0x7ff4cc0ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:13.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 -- 172.21.15.38:0/3363719011 shutdown_connections 2024-03-26T06:48:13.290 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.285+0000 7ff4d2854640 1 -- 172.21.15.38:0/3363719011 wait complete. 2024-03-26T06:48:13.299 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-26T06:48:13.408 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.402+0000 7fb8930ea640 1 -- 172.21.15.38:0/997419227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb88c100d80 msgr2=0x7fb88c104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.402+0000 7fb8930ea640 1 --2- 172.21.15.38:0/997419227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb88c100d80 0x7fb88c104dc0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fb880009900 tx=0x7fb88002bc30 comp rx=0 tx=0).stop 2024-03-26T06:48:13.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.402+0000 7fb8930ea640 1 -- 172.21.15.38:0/997419227 shutdown_connections 2024-03-26T06:48:13.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.402+0000 7fb8930ea640 1 --2- 172.21.15.38:0/997419227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb88c1054a0 0x7fb88c10d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.409 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.402+0000 7fb8930ea640 1 --2- 172.21.15.38:0/997419227 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb88c100d80 0x7fb88c104dc0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.402+0000 7fb8930ea640 1 --2- 172.21.15.38:0/997419227 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c1003d0 0x7fb88c1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.402+0000 7fb8930ea640 1 -- 172.21.15.38:0/997419227 >> 172.21.15.38:0/997419227 conn(0x7fb88c0772f0 msgr2=0x7fb88c077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:13.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.402+0000 7fb8930ea640 1 -- 172.21.15.38:0/997419227 shutdown_connections 2024-03-26T06:48:13.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.402+0000 7fb8930ea640 1 -- 172.21.15.38:0/997419227 wait complete. 2024-03-26T06:48:13.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.403+0000 7fb8930ea640 1 Processor -- start 2024-03-26T06:48:13.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.403+0000 7fb8930ea640 1 -- start start 2024-03-26T06:48:13.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.403+0000 7fb8930ea640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb88c1003d0 0x7fb88c101e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.410 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.403+0000 7fb8930ea640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c1054a0 0x7fb88c1023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.403+0000 7fb8930ea640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb88c1028e0 0x7fb88c1a5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.403+0000 7fb8930ea640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c110530 con 0x7fb88c1028e0 2024-03-26T06:48:13.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.403+0000 7fb8930ea640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c1103b0 con 0x7fb88c1003d0 2024-03-26T06:48:13.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.403+0000 7fb8930ea640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88c1106b0 con 0x7fb88c1054a0 2024-03-26T06:48:13.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.403+0000 7fb891687640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb88c1028e0 0x7fb88c1a5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:13.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.403+0000 7fb891687640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb88c1028e0 0x7fb88c1a5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54280/0 (socket says 172.21.15.38:54280) 2024-03-26T06:48:13.411 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.403+0000 7fb891687640 1 -- 172.21.15.38:0/1294566224 learned_addr learned my addr 172.21.15.38:0/1294566224 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:13.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb890e86640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb88c1003d0 0x7fb88c101e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:13.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb88bfff640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c1054a0 0x7fb88c1023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:13.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb891687640 1 -- 172.21.15.38:0/1294566224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c1054a0 msgr2=0x7fb88c1023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb891687640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c1054a0 0x7fb88c1023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb891687640 1 -- 172.21.15.38:0/1294566224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb88c1003d0 msgr2=0x7fb88c101e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb891687640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb88c1003d0 0x7fb88c101e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb891687640 1 -- 172.21.15.38:0/1294566224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88c1a6450 con 0x7fb88c1028e0 2024-03-26T06:48:13.412 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb891687640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb88c1028e0 0x7fb88c1a5d50 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb87c00e9b0 tx=0x7fb87c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:13.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb889ffb640 1 -- 172.21.15.38:0/1294566224 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c00cd80 con 0x7fb88c1028e0 2024-03-26T06:48:13.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb889ffb640 1 -- 172.21.15.38:0/1294566224 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb87c0035f0 con 0x7fb88c1028e0 2024-03-26T06:48:13.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb889ffb640 1 -- 172.21.15.38:0/1294566224 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87c010670 con 0x7fb88c1028e0 2024-03-26T06:48:13.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb8930ea640 1 -- 172.21.15.38:0/1294566224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88c1a6740 con 0x7fb88c1028e0 2024-03-26T06:48:13.413 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.404+0000 7fb8930ea640 1 -- 172.21.15.38:0/1294566224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88c1a6c60 con 0x7fb88c1028e0 2024-03-26T06:48:13.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.405+0000 7fb8930ea640 1 -- 172.21.15.38:0/1294566224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88c0680e0 con 0x7fb88c1028e0 2024-03-26T06:48:13.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.406+0000 7fb889ffb640 1 -- 172.21.15.38:0/1294566224 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb87c0037a0 con 0x7fb88c1028e0 2024-03-26T06:48:13.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.407+0000 7fb889ffb640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb8740a2a20 0x7fb8740a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.414 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.407+0000 7fb889ffb640 1 -- 172.21.15.38:0/1294566224 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8629+0+0 (secure 0 0 0) 0x7fb87c0cf140 con 0x7fb88c1028e0 2024-03-26T06:48:13.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.407+0000 7fb890e86640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb8740a2a20 0x7fb8740a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:13.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.407+0000 7fb890e86640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb8740a2a20 0x7fb8740a4ee0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb884005fd0 tx=0x7fb884009870 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:13.415 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.411+0000 7fb889ffb640 1 -- 172.21.15.38:0/1294566224 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb87c09bed0 con 0x7fb88c1028e0 2024-03-26T06:48:13.597 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.594+0000 7fb8930ea640 1 -- 172.21.15.38:0/1294566224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb88c109700 con 0x7fb88c1028e0 2024-03-26T06:48:13.598 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.594+0000 7fb889ffb640 1 -- 172.21.15.38:0/1294566224 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb87c09b7d0 con 0x7fb88c1028e0 2024-03-26T06:48:13.598 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2024-03-26T06:48:13.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 -- 172.21.15.38:0/1294566224 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb8740a2a20 msgr2=0x7fb8740a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.600 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb8740a2a20 0x7fb8740a4ee0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb884005fd0 tx=0x7fb884009870 comp rx=0 tx=0).stop 2024-03-26T06:48:13.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 -- 172.21.15.38:0/1294566224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb88c1028e0 msgr2=0x7fb88c1a5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb88c1028e0 0x7fb88c1a5d50 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fb87c00e9b0 tx=0x7fb87c00ee70 comp rx=0 tx=0).stop 2024-03-26T06:48:13.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 -- 172.21.15.38:0/1294566224 shutdown_connections 2024-03-26T06:48:13.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fb8740a2a20 0x7fb8740a4ee0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb88c1028e0 0x7fb88c1a5d50 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fb88c1054a0 0x7fb88c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.601 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 --2- 172.21.15.38:0/1294566224 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb88c1003d0 0x7fb88c101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 -- 172.21.15.38:0/1294566224 >> 172.21.15.38:0/1294566224 conn(0x7fb88c0772f0 msgr2=0x7fb88c0687f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:13.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 -- 172.21.15.38:0/1294566224 shutdown_connections 2024-03-26T06:48:13.602 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.597+0000 7fb8930ea640 1 -- 172.21.15.38:0/1294566224 wait complete. 2024-03-26T06:48:13.612 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-26T06:48:13.719 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.713+0000 7fd3e5c14640 1 -- 172.21.15.38:0/794835289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3e0101ea0 msgr2=0x7fd3e0105ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.713+0000 7fd3e5c14640 1 --2- 172.21.15.38:0/794835289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3e0101ea0 0x7fd3e0105ee0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009900 tx=0x7fd3d402bc30 comp rx=0 tx=0).stop 2024-03-26T06:48:13.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.713+0000 7fd3e5c14640 1 -- 172.21.15.38:0/794835289 shutdown_connections 2024-03-26T06:48:13.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.713+0000 7fd3e5c14640 1 --2- 172.21.15.38:0/794835289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e01065c0 0x7fd3e010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.713+0000 7fd3e5c14640 1 --2- 172.21.15.38:0/794835289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3e0101ea0 0x7fd3e0105ee0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.713+0000 7fd3e5c14640 1 --2- 172.21.15.38:0/794835289 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3e01014f0 0x7fd3e01018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.720 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.713+0000 7fd3e5c14640 1 -- 172.21.15.38:0/794835289 >> 172.21.15.38:0/794835289 conn(0x7fd3e00786f0 msgr2=0x7fd3e0078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:13.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.713+0000 7fd3e5c14640 1 -- 172.21.15.38:0/794835289 shutdown_connections 2024-03-26T06:48:13.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.713+0000 7fd3e5c14640 1 -- 172.21.15.38:0/794835289 wait complete. 2024-03-26T06:48:13.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.713+0000 7fd3e5c14640 1 Processor -- start 2024-03-26T06:48:13.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3e5c14640 1 -- start start 2024-03-26T06:48:13.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3e5c14640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3e01014f0 0x7fd3e019d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.721 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3e5c14640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e01065c0 0x7fd3e019dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3e5c14640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3e01979d0 0x7fd3e0197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3e5c14640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e0111650 con 0x7fd3e01014f0 2024-03-26T06:48:13.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3e5c14640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e01114d0 con 0x7fd3e01065c0 2024-03-26T06:48:13.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3e5c14640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e01117d0 con 0x7fd3e01979d0 2024-03-26T06:48:13.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3e01014f0 0x7fd3e019d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:13.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3df7fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3e01014f0 0x7fd3e019d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:54286/0 (socket says 172.21.15.38:54286) 2024-03-26T06:48:13.722 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3df7fe640 1 -- 172.21.15.38:0/3943594852 learned_addr learned my addr 172.21.15.38:0/3943594852 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:13.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3deffd640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e01065c0 0x7fd3e019dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:13.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3df7fe640 1 -- 172.21.15.38:0/3943594852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3e01979d0 msgr2=0x7fd3e0197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3df7fe640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3e01979d0 0x7fd3e0197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3df7fe640 1 -- 172.21.15.38:0/3943594852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e01065c0 msgr2=0x7fd3e019dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3df7fe640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e01065c0 0x7fd3e019dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3df7fe640 1 -- 172.21.15.38:0/3943594852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e01984f0 con 0x7fd3e01014f0 2024-03-26T06:48:13.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3df7fe640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3e01014f0 0x7fd3e019d820 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000ea80 tx=0x7fd3d000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:13.723 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3dcff9640 1 -- 172.21.15.38:0/3943594852 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d000ce40 con 0x7fd3e01014f0 2024-03-26T06:48:13.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.714+0000 7fd3dcff9640 1 -- 172.21.15.38:0/3943594852 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3d0005020 con 0x7fd3e01014f0 2024-03-26T06:48:13.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.715+0000 7fd3e5c14640 1 -- 172.21.15.38:0/3943594852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e01987e0 con 0x7fd3e01014f0 2024-03-26T06:48:13.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.715+0000 7fd3dcff9640 1 -- 172.21.15.38:0/3943594852 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0011760 con 0x7fd3e01014f0 2024-03-26T06:48:13.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.715+0000 7fd3e5c14640 1 -- 172.21.15.38:0/3943594852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e01a49e0 con 0x7fd3e01014f0 2024-03-26T06:48:13.724 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.716+0000 7fd3e5c14640 1 -- 172.21.15.38:0/3943594852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e0068530 con 0x7fd3e01014f0 2024-03-26T06:48:13.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.717+0000 7fd3dcff9640 1 -- 172.21.15.38:0/3943594852 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd3d00051d0 con 0x7fd3e01014f0 2024-03-26T06:48:13.725 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.717+0000 7fd3dcff9640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd3c80a2970 0x7fd3c80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:13.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.717+0000 7fd3deffd640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd3c80a2970 0x7fd3c80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:13.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.718+0000 7fd3dcff9640 1 -- 172.21.15.38:0/3943594852 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8629+0+0 (secure 0 0 0) 0x7fd3d00cf530 con 0x7fd3e01014f0 2024-03-26T06:48:13.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.718+0000 7fd3deffd640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd3c80a2970 0x7fd3c80a4e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009230 tx=0x7fd3d4005fd0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:13.726 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.722+0000 7fd3dcff9640 1 -- 172.21.15.38:0/3943594852 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd3d0016030 con 0x7fd3e01014f0 2024-03-26T06:48:13.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:13 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3678414198' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:48:13.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:13 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1103009775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:48:13.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:13 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3363719011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:48:13.885 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:13 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3678414198' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:48:13.885 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:13 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1103009775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:48:13.885 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:13 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3363719011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:48:13.895 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:13 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3678414198' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:48:13.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:13 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1103009775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T06:48:13.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:13 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3363719011' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-26T06:48:13.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.919+0000 7fd3e5c14640 1 -- 172.21.15.38:0/3943594852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd3e0101ea0 con 0x7fd3e01014f0 2024-03-26T06:48:13.923 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.920+0000 7fd3dcff9640 1 -- 172.21.15.38:0/3943594852 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 17 entries v99) v1 ==== 81+0+1004 (secure 0 0 0) 0x7fd3d009c220 con 0x7fd3e01014f0 2024-03-26T06:48:13.924 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/3896600378 2024-03-27T06:28:26.137775+0000 2024-03-26T06:48:13.924 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/1985135432 2024-03-27T06:27:59.817781+0000 2024-03-26T06:48:13.924 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/3929636377 2024-03-27T06:27:59.817781+0000 2024-03-26T06:48:13.924 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2512207794 2024-03-27T06:27:59.817781+0000 2024-03-26T06:48:13.925 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2579773966 2024-03-27T06:27:59.817781+0000 2024-03-26T06:48:13.925 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/1523129198 2024-03-27T06:28:26.137775+0000 2024-03-26T06:48:13.925 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/2716021457 2024-03-27T06:37:32.618701+0000 2024-03-26T06:48:13.925 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2750459261 2024-03-27T06:37:32.618701+0000 2024-03-26T06:48:13.925 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/4246298230 2024-03-27T06:28:26.137775+0000 2024-03-26T06:48:13.925 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/4246298230 2024-03-27T06:28:26.137775+0000 2024-03-26T06:48:13.925 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/2527804067 2024-03-27T06:28:26.137775+0000 2024-03-26T06:48:13.926 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/1985135432 2024-03-27T06:27:59.817781+0000 2024-03-26T06:48:13.926 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/1675065388 2024-03-27T06:37:32.618701+0000 2024-03-26T06:48:13.926 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/4027757314 2024-03-27T06:37:32.618701+0000 2024-03-26T06:48:13.926 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/2716021457 2024-03-27T06:37:32.618701+0000 2024-03-26T06:48:13.926 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/3320689790 2024-03-27T06:37:32.618701+0000 2024-03-26T06:48:13.926 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/3570258347 2024-03-27T06:37:32.618701+0000 2024-03-26T06:48:13.927 INFO:teuthology.orchestra.run.smithi038.stderr:listed 17 entries 2024-03-26T06:48:13.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 -- 172.21.15.38:0/3943594852 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd3c80a2970 msgr2=0x7fd3c80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.927 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd3c80a2970 0x7fd3c80a4e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd3d4009230 tx=0x7fd3d4005fd0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 -- 172.21.15.38:0/3943594852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3e01014f0 msgr2=0x7fd3e019d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3e01014f0 0x7fd3e019d820 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000ea80 tx=0x7fd3d000ef40 comp rx=0 tx=0).stop 2024-03-26T06:48:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 -- 172.21.15.38:0/3943594852 shutdown_connections 2024-03-26T06:48:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fd3c80a2970 0x7fd3c80a4e30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fd3e01979d0 0x7fd3e0197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e01065c0 0x7fd3e019dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 --2- 172.21.15.38:0/3943594852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3e01014f0 0x7fd3e019d820 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 -- 172.21.15.38:0/3943594852 >> 172.21.15.38:0/3943594852 conn(0x7fd3e00786f0 msgr2=0x7fd3e00ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 -- 172.21.15.38:0/3943594852 shutdown_connections 2024-03-26T06:48:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:13.922+0000 7fd3e5c14640 1 -- 172.21.15.38:0/3943594852 wait complete. 2024-03-26T06:48:13.937 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-26T06:48:13.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:48:13.937 DEBUG:teuthology.orchestra.run.smithi038:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-26T06:48:14.746 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T06:48:14.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:48:14.747 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-26T06:48:14.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:48:14.783 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2024-03-26T06:48:14.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:14 smithi038 ceph-mon[32265]: pgmap v333: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-03-26T06:48:14.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:14 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1294566224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:48:14.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:14 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3943594852' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T06:48:14.839 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-26T06:48:14.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:48:14.840 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-26T06:48:14.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:14 smithi121 ceph-mon[36349]: pgmap v333: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-03-26T06:48:14.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:14 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1294566224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:48:14.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:14 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3943594852' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T06:48:14.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-26T06:48:14.908 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link delete ceph-brx 2024-03-26T06:48:14.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:14 smithi170 ceph-mon[36912]: pgmap v333: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-03-26T06:48:14.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:14 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1294566224' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-26T06:48:14.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:14 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3943594852' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T06:48:15.007 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T06:48:15.023 INFO:tasks.cephadm:Teardown begin 2024-03-26T06:48:15.023 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-26T06:48:15.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:48:15.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.123+0000 7f3619724640 1 -- 172.21.15.38:0/3625052249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3614100db0 msgr2=0x7f3614104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.123+0000 7f3619724640 1 --2- 172.21.15.38:0/3625052249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3614100db0 0x7f3614104df0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f3608009900 tx=0x7f360802bc30 comp rx=0 tx=0).stop 2024-03-26T06:48:15.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.123+0000 7f3619724640 1 -- 172.21.15.38:0/3625052249 shutdown_connections 2024-03-26T06:48:15.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.123+0000 7f3619724640 1 --2- 172.21.15.38:0/3625052249 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36141054d0 0x7f361410d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.123+0000 7f3619724640 1 --2- 172.21.15.38:0/3625052249 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3614100db0 0x7f3614104df0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.129 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.123+0000 7f3619724640 1 --2- 172.21.15.38:0/3625052249 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3614100400 0x7f36141007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.123+0000 7f3619724640 1 -- 172.21.15.38:0/3625052249 >> 172.21.15.38:0/3625052249 conn(0x7f3614076570 msgr2=0x7f3614076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:15.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.123+0000 7f3619724640 1 -- 172.21.15.38:0/3625052249 shutdown_connections 2024-03-26T06:48:15.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.123+0000 7f3619724640 1 -- 172.21.15.38:0/3625052249 wait complete. 2024-03-26T06:48:15.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f3619724640 1 Processor -- start 2024-03-26T06:48:15.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f3619724640 1 -- start start 2024-03-26T06:48:15.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f3619724640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3614100400 0x7f3614197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f3619724640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36141054d0 0x7f3614198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.130 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f3619724640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f361419c520 0x7f361419c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f3619724640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3614110560 con 0x7f361419c520 2024-03-26T06:48:15.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f3619724640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36141103e0 con 0x7f36141054d0 2024-03-26T06:48:15.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f3619724640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36141106e0 con 0x7f3614100400 2024-03-26T06:48:15.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f36137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f361419c520 0x7f361419c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f36137fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f361419c520 0x7f361419c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47126/0 (socket says 172.21.15.38:47126) 2024-03-26T06:48:15.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f3612ffd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3614100400 0x7f3614197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f36137fe640 1 -- 172.21.15.38:0/1099964649 learned_addr learned my addr 172.21.15.38:0/1099964649 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:15.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.124+0000 7f36127fc640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36141054d0 0x7f3614198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.131 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f36137fe640 1 -- 172.21.15.38:0/1099964649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3614100400 msgr2=0x7f3614197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f36137fe640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3614100400 0x7f3614197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f36137fe640 1 -- 172.21.15.38:0/1099964649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36141054d0 msgr2=0x7f3614198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f36137fe640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36141054d0 0x7f3614198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f36137fe640 1 -- 172.21.15.38:0/1099964649 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361419d010 con 0x7f361419c520 2024-03-26T06:48:15.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f36127fc640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36141054d0 0x7f3614198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T06:48:15.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f36137fe640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f361419c520 0x7f361419c9d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f360c00efd0 tx=0x7f360c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:15.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f35fbfff640 1 -- 172.21.15.38:0/1099964649 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c003b70 con 0x7f361419c520 2024-03-26T06:48:15.132 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f3619724640 1 -- 172.21.15.38:0/1099964649 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361419d300 con 0x7f361419c520 2024-03-26T06:48:15.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f35fbfff640 1 -- 172.21.15.38:0/1099964649 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f360c003d10 con 0x7f361419c520 2024-03-26T06:48:15.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f35fbfff640 1 -- 172.21.15.38:0/1099964649 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360c0109f0 con 0x7f361419c520 2024-03-26T06:48:15.133 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.125+0000 7f3619724640 1 -- 172.21.15.38:0/1099964649 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36141a4a70 con 0x7f361419c520 2024-03-26T06:48:15.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.126+0000 7f3619724640 1 -- 172.21.15.38:0/1099964649 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36141018d0 con 0x7f361419c520 2024-03-26T06:48:15.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.127+0000 7f35fbfff640 1 -- 172.21.15.38:0/1099964649 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f360c004020 con 0x7f361419c520 2024-03-26T06:48:15.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.127+0000 7f35fbfff640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f35fc0a2970 0x7f35fc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.128+0000 7f3612ffd640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f35fc0a2970 0x7f35fc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.128+0000 7f35fbfff640 1 -- 172.21.15.38:0/1099964649 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8629+0+0 (secure 0 0 0) 0x7f360c0cf3b0 con 0x7f361419c520 2024-03-26T06:48:15.134 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.128+0000 7f3612ffd640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f35fc0a2970 0x7f35fc0a4e30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3604005fd0 tx=0x7f360400b390 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:15.135 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.132+0000 7f35fbfff640 1 -- 172.21.15.38:0/1099964649 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f360c09c040 con 0x7f361419c520 2024-03-26T06:48:15.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.359+0000 7f3619724640 1 -- 172.21.15.38:0/1099964649 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3614109710 con 0x7f361419c520 2024-03-26T06:48:15.363 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.360+0000 7f35fbfff640 1 -- 172.21.15.38:0/1099964649 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+11346 (secure 0 0 0) 0x7f360c09b940 con 0x7f361419c520 2024-03-26T06:48:15.364 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:48:15.364 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/357411663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":357411663},{"type":"v1","addr":"172.21.15.38:6839","nonce":357411663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/654319512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":654319512},{"type":"v1","addr":"172.21.15.38:6841","nonce":654319512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-26T06:35:26.477444+0000","modified":"2024-03-26T06:41:56.894438+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24502,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/359437185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":359437185},{"type":"v1","addr":"172.21.15.121:6839","nonce":359437185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24991":{"gid":24991,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1404647281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1404647281},{"type":"v1","addr":"172.21.15.170:6837","nonce":1404647281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-26T06:48:15.364 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 30 2024-03-26T06:48:15.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 -- 172.21.15.38:0/1099964649 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f35fc0a2970 msgr2=0x7f35fc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f35fc0a2970 0x7f35fc0a4e30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3604005fd0 tx=0x7f360400b390 comp rx=0 tx=0).stop 2024-03-26T06:48:15.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 -- 172.21.15.38:0/1099964649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f361419c520 msgr2=0x7f361419c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f361419c520 0x7f361419c9d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f360c00efd0 tx=0x7f360c00c6a0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 -- 172.21.15.38:0/1099964649 shutdown_connections 2024-03-26T06:48:15.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f35fc0a2970 0x7f35fc0a4e30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.366 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f361419c520 0x7f361419c9d0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f36141054d0 0x7f3614198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 --2- 172.21.15.38:0/1099964649 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f3614100400 0x7f3614197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 -- 172.21.15.38:0/1099964649 >> 172.21.15.38:0/1099964649 conn(0x7f3614076570 msgr2=0x7f3614069c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:15.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 -- 172.21.15.38:0/1099964649 shutdown_connections 2024-03-26T06:48:15.367 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.362+0000 7f3619724640 1 -- 172.21.15.38:0/1099964649 wait complete. 2024-03-26T06:48:15.377 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-26T06:48:15.482 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.477+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1730073645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2e81054d0 msgr2=0x7fe2e810d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.477+0000 7fe2ee3cd640 1 --2- 172.21.15.38:0/1730073645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2e81054d0 0x7fe2e810d0a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0009ca0 tx=0x7fe2e002c0f0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.477+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1730073645 shutdown_connections 2024-03-26T06:48:15.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.477+0000 7fe2ee3cd640 1 --2- 172.21.15.38:0/1730073645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2e81054d0 0x7fe2e810d0a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.477+0000 7fe2ee3cd640 1 --2- 172.21.15.38:0/1730073645 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2e8100db0 0x7fe2e8104df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.477+0000 7fe2ee3cd640 1 --2- 172.21.15.38:0/1730073645 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2e8100400 0x7fe2e81007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.483 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.477+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1730073645 >> 172.21.15.38:0/1730073645 conn(0x7fe2e8076570 msgr2=0x7fe2e8076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.477+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1730073645 shutdown_connections 2024-03-26T06:48:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.477+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1730073645 wait complete. 2024-03-26T06:48:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2ee3cd640 1 Processor -- start 2024-03-26T06:48:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2ee3cd640 1 -- start start 2024-03-26T06:48:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2ee3cd640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2e8100400 0x7fe2e81988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2ee3cd640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2e81054d0 0x7fe2e8198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2ee3cd640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2e81993a0 0x7fe2e81a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2ee3cd640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e8110570 con 0x7fe2e81054d0 2024-03-26T06:48:15.484 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2ee3cd640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e81103f0 con 0x7fe2e81993a0 2024-03-26T06:48:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2ee3cd640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e81106f0 con 0x7fe2e8100400 2024-03-26T06:48:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2e77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2e81054d0 0x7fe2e8198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2e77fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2e81054d0 0x7fe2e8198e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47146/0 (socket says 172.21.15.38:47146) 2024-03-26T06:48:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2e77fe640 1 -- 172.21.15.38:0/1373749581 learned_addr learned my addr 172.21.15.38:0/1373749581 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2ec96a640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2e81993a0 0x7fe2e81a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2e7fff640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2e8100400 0x7fe2e81988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2e77fe640 1 -- 172.21.15.38:0/1373749581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2e8100400 msgr2=0x7fe2e81988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.485 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2e77fe640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2e8100400 0x7fe2e81988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2e77fe640 1 -- 172.21.15.38:0/1373749581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2e81993a0 msgr2=0x7fe2e81a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2e77fe640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2e81993a0 0x7fe2e81a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.478+0000 7fe2e77fe640 1 -- 172.21.15.38:0/1373749581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e81a4120 con 0x7fe2e81054d0 2024-03-26T06:48:15.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.479+0000 7fe2e77fe640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2e81054d0 0x7fe2e8198e10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800d770 tx=0x7fe2d800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:15.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.479+0000 7fe2e57fa640 1 -- 172.21.15.38:0/1373749581 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8017070 con 0x7fe2e81054d0 2024-03-26T06:48:15.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.479+0000 7fe2e57fa640 1 -- 172.21.15.38:0/1373749581 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe2d8009c30 con 0x7fe2e81054d0 2024-03-26T06:48:15.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.479+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1373749581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e81a4410 con 0x7fe2e81054d0 2024-03-26T06:48:15.486 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.479+0000 7fe2e57fa640 1 -- 172.21.15.38:0/1373749581 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d800f620 con 0x7fe2e81054d0 2024-03-26T06:48:15.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.479+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1373749581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e81a4960 con 0x7fe2e81054d0 2024-03-26T06:48:15.487 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.480+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1373749581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e81018d0 con 0x7fe2e81054d0 2024-03-26T06:48:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.481+0000 7fe2e57fa640 1 -- 172.21.15.38:0/1373749581 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fe2d801a070 con 0x7fe2e81054d0 2024-03-26T06:48:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.481+0000 7fe2e57fa640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2d00a2970 0x7fe2d00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.482+0000 7fe2e7fff640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2d00a2970 0x7fe2d00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.482+0000 7fe2e57fa640 1 -- 172.21.15.38:0/1373749581 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8629+0+0 (secure 0 0 0) 0x7fe2d80ce960 con 0x7fe2e81054d0 2024-03-26T06:48:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.482+0000 7fe2e7fff640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2d00a2970 0x7fe2d00a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00ba40 tx=0x7fe2dc005cc0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.486+0000 7fe2e57fa640 1 -- 172.21.15.38:0/1373749581 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe2d8009de0 con 0x7fe2e81054d0 2024-03-26T06:48:15.713 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.710+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1373749581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2e8108700 con 0x7fe2e81054d0 2024-03-26T06:48:15.714 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.711+0000 7fe2e57fa640 1 -- 172.21.15.38:0/1373749581 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+11346 (secure 0 0 0) 0x7fe2d809b650 con 0x7fe2e81054d0 2024-03-26T06:48:15.714 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:48:15.715 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/4105682236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":4105682236},{"type":"v1","addr":"172.21.15.38:6835","nonce":4105682236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/3363372565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":3363372565},{"type":"v1","addr":"172.21.15.38:6837","nonce":3363372565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14628,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/357411663","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":357411663},{"type":"v1","addr":"172.21.15.38:6839","nonce":357411663}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14640,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/654319512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":654319512},{"type":"v1","addr":"172.21.15.38:6841","nonce":654319512}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24412,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.170:6833/1336487813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6832","nonce":1336487813},{"type":"v1","addr":"172.21.15.170:6833","nonce":1336487813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6833/2121798105","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2121798105},{"type":"v1","addr":"172.21.15.121:6833","nonce":2121798105}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-26T06:35:26.477444+0000","modified":"2024-03-26T06:41:56.894438+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24539,"mds_1":24502,"mds_2":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24460":{"gid":24460,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6835/2374885191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6834","nonce":2374885191},{"type":"v1","addr":"172.21.15.170:6835","nonce":2374885191}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.121:6835/3384748966","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6834","nonce":3384748966},{"type":"v1","addr":"172.21.15.121:6835","nonce":3384748966}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"h","rank":2,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.121:6837/1114423776","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6836","nonce":1114423776},{"type":"v1","addr":"172.21.15.121:6837","nonce":1114423776}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24502":{"gid":24502,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.170:6839/2204378068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6838","nonce":2204378068},{"type":"v1","addr":"172.21.15.170:6839","nonce":2204378068}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24539":{"gid":24539,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.121:6839/359437185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6838","nonce":359437185},{"type":"v1","addr":"172.21.15.121:6839","nonce":359437185}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24991":{"gid":24991,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.170:6837/1404647281","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6836","nonce":1404647281},{"type":"v1","addr":"172.21.15.170:6837","nonce":1404647281}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-26T06:48:15.715 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 30 2024-03-26T06:48:15.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1373749581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2d00a2970 msgr2=0x7fe2d00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2d00a2970 0x7fe2d00a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe2dc00ba40 tx=0x7fe2dc005cc0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1373749581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2e81054d0 msgr2=0x7fe2e8198e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2e81054d0 0x7fe2e8198e10 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe2d800d770 tx=0x7fe2d800dc30 comp rx=0 tx=0).stop 2024-03-26T06:48:15.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1373749581 shutdown_connections 2024-03-26T06:48:15.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fe2d00a2970 0x7fe2d00a4e30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2e81993a0 0x7fe2e81a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.717 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe2e81054d0 0x7fe2e8198e10 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 --2- 172.21.15.38:0/1373749581 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fe2e8100400 0x7fe2e81988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1373749581 >> 172.21.15.38:0/1373749581 conn(0x7fe2e8076570 msgr2=0x7fe2e80fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:15.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1373749581 shutdown_connections 2024-03-26T06:48:15.718 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.713+0000 7fe2ee3cd640 1 -- 172.21.15.38:0/1373749581 wait complete. 2024-03-26T06:48:15.727 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:15 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1099964649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:48:15.728 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-26T06:48:15.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.828+0000 7f5b79553640 1 -- 172.21.15.38:0/3629972179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74106610 msgr2=0x7f5b7410e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.833 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.828+0000 7f5b79553640 1 --2- 172.21.15.38:0/3629972179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74106610 0x7f5b7410e1e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c009ca0 tx=0x7f5b6c02c0f0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.828+0000 7f5b79553640 1 -- 172.21.15.38:0/3629972179 shutdown_connections 2024-03-26T06:48:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.828+0000 7f5b79553640 1 --2- 172.21.15.38:0/3629972179 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74106610 0x7f5b7410e1e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.828+0000 7f5b79553640 1 --2- 172.21.15.38:0/3629972179 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b7406a5c0 0x7f5b74105f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.828+0000 7f5b79553640 1 --2- 172.21.15.38:0/3629972179 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b74069c10 0x7f5b74069ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.828+0000 7f5b79553640 1 -- 172.21.15.38:0/3629972179 >> 172.21.15.38:0/3629972179 conn(0x7f5b74077290 msgr2=0x7f5b740776a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:15.834 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.828+0000 7f5b79553640 1 -- 172.21.15.38:0/3629972179 shutdown_connections 2024-03-26T06:48:15.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.828+0000 7f5b79553640 1 -- 172.21.15.38:0/3629972179 wait complete. 2024-03-26T06:48:15.838 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.828+0000 7f5b79553640 1 Processor -- start 2024-03-26T06:48:15.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b79553640 1 -- start start 2024-03-26T06:48:15.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b79553640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b74069c10 0x7f5b7419d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b79553640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b74106610 0x7f5b7419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b79553640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74197980 0x7f5b74197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b79553640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b741116c0 con 0x7f5b74197980 2024-03-26T06:48:15.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b79553640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b74111540 con 0x7f5b74069c10 2024-03-26T06:48:15.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b79553640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b74111840 con 0x7f5b74106610 2024-03-26T06:48:15.839 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b737fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74197980 0x7f5b74197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b737fe640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74197980 0x7f5b74197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47156/0 (socket says 172.21.15.38:47156) 2024-03-26T06:48:15.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b737fe640 1 -- 172.21.15.38:0/1825104563 learned_addr learned my addr 172.21.15.38:0/1825104563 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:15.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b72ffd640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b74069c10 0x7f5b7419d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b727fc640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b74106610 0x7f5b7419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b737fe640 1 -- 172.21.15.38:0/1825104563 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b74106610 msgr2=0x7f5b7419dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b737fe640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b74106610 0x7f5b7419dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b737fe640 1 -- 172.21.15.38:0/1825104563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b74069c10 msgr2=0x7f5b7419d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:15.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b737fe640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b74069c10 0x7f5b7419d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:15.840 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b737fe640 1 -- 172.21.15.38:0/1825104563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b741984a0 con 0x7f5b74197980 2024-03-26T06:48:15.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.829+0000 7f5b737fe640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74197980 0x7f5b74197e30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c02c700 tx=0x7f5b6c032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:15.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.830+0000 7f5b5bfff640 1 -- 172.21.15.38:0/1825104563 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c039040 con 0x7f5b74197980 2024-03-26T06:48:15.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.830+0000 7f5b5bfff640 1 -- 172.21.15.38:0/1825104563 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b6c044070 con 0x7f5b74197980 2024-03-26T06:48:15.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.830+0000 7f5b79553640 1 -- 172.21.15.38:0/1825104563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b74198730 con 0x7f5b74197980 2024-03-26T06:48:15.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.830+0000 7f5b5bfff640 1 -- 172.21.15.38:0/1825104563 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6c0042a0 con 0x7f5b74197980 2024-03-26T06:48:15.841 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.830+0000 7f5b79553640 1 -- 172.21.15.38:0/1825104563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b741a4980 con 0x7f5b74197980 2024-03-26T06:48:15.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.831+0000 7f5b79553640 1 -- 172.21.15.38:0/1825104563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b74102a10 con 0x7f5b74197980 2024-03-26T06:48:15.842 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.832+0000 7f5b5bfff640 1 -- 172.21.15.38:0/1825104563 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5b6c004440 con 0x7f5b74197980 2024-03-26T06:48:15.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.832+0000 7f5b5bfff640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f5b5c0a29f0 0x7f5b5c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:15.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.832+0000 7f5b72ffd640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f5b5c0a29f0 0x7f5b5c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:15.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.832+0000 7f5b5bfff640 1 -- 172.21.15.38:0/1825104563 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8629+0+0 (secure 0 0 0) 0x7f5b6c0f1660 con 0x7f5b74197980 2024-03-26T06:48:15.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.833+0000 7f5b72ffd640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f5b5c0a29f0 0x7f5b5c0a4eb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5b74198ac0 tx=0x7f5b6400b390 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:15.843 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:15.836+0000 7f5b5bfff640 1 -- 172.21.15.38:0/1825104563 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5b6c0be460 con 0x7f5b74197980 2024-03-26T06:48:15.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:15 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1099964649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:48:15.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:15 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1099964649' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:48:16.024 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.020+0000 7f5b79553640 1 -- 172.21.15.38:0/1825104563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b74068530 con 0x7f5b74197980 2024-03-26T06:48:16.025 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.022+0000 7f5b5bfff640 1 -- 172.21.15.38:0/1825104563 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+23231 (secure 0 0 0) 0x7f5b74068530 con 0x7f5b74197980 2024-03-26T06:48:16.026 INFO:teuthology.orchestra.run.smithi038.stdout: 2024-03-26T06:48:16.026 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":99,"fsid":"c2824556-eb39-11ee-95d0-87774f69a715","created":"2024-03-26T06:27:32.376242+0000","modified":"2024-03-26T06:48:07.543242+0000","last_up_change":"2024-03-26T06:34:36.019682+0000","last_in_change":"2024-03-26T06:34:16.083412+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T06:30:52.485897+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-26T06:35:24.501259+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-26T06:35:25.675299+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":99,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7ec203cd-dc26-4133-83b3-de1d661ddd0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6803","nonce":1449059243}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6805","nonce":1449059243}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6809","nonce":1449059243}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1449059243},{"type":"v1","addr":"172.21.15.38:6807","nonce":1449059243}]},"public_addr":"172.21.15.38:6803/1449059243","cluster_addr":"172.21.15.38:6805/1449059243","heartbeat_back_addr":"172.21.15.38:6809/1449059243","heartbeat_front_addr":"172.21.15.38:6807/1449059243","state":["exists","up"]},{"osd":1,"uuid":"3ab1d9ef-d56f-49fa-a7e7-833d1c175f8f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6801","nonce":3506119522}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6803","nonce":3506119522}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6807","nonce":3506119522}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":3506119522},{"type":"v1","addr":"172.21.15.121:6805","nonce":3506119522}]},"public_addr":"172.21.15.121:6801/3506119522","cluster_addr":"172.21.15.121:6803/3506119522","heartbeat_back_addr":"172.21.15.121:6807/3506119522","heartbeat_front_addr":"172.21.15.121:6805/3506119522","state":["exists","up"]},{"osd":2,"uuid":"09ebac77-3d71-4e0a-b07a-6797adda9dd0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6800","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6801","nonce":1189530871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6802","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6803","nonce":1189530871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6806","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6807","nonce":1189530871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6804","nonce":1189530871},{"type":"v1","addr":"172.21.15.170:6805","nonce":1189530871}]},"public_addr":"172.21.15.170:6801/1189530871","cluster_addr":"172.21.15.170:6803/1189530871","heartbeat_back_addr":"172.21.15.170:6807/1189530871","heartbeat_front_addr":"172.21.15.170:6805/1189530871","state":["exists","up"]},{"osd":3,"uuid":"05893f6a-6566-412d-8341-72c9d577310b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6811","nonce":3343509291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6813","nonce":3343509291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6817","nonce":3343509291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":3343509291},{"type":"v1","addr":"172.21.15.38:6815","nonce":3343509291}]},"public_addr":"172.21.15.38:6811/3343509291","cluster_addr":"172.21.15.38:6813/3343509291","heartbeat_back_addr":"172.21.15.38:6817/3343509291","heartbeat_front_addr":"172.21.15.38:6815/3343509291","state":["exists","up"]},{"osd":4,"uuid":"14ae6f68-9b15-4098-8bea-4c1688b4a0fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6809","nonce":1471300176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6811","nonce":1471300176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6815","nonce":1471300176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":1471300176},{"type":"v1","addr":"172.21.15.121:6813","nonce":1471300176}]},"public_addr":"172.21.15.121:6809/1471300176","cluster_addr":"172.21.15.121:6811/1471300176","heartbeat_back_addr":"172.21.15.121:6815/1471300176","heartbeat_front_addr":"172.21.15.121:6813/1471300176","state":["exists","up"]},{"osd":5,"uuid":"8bb8ecf7-e48a-4e5a-b313-f84660cf29f6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6808","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6809","nonce":718927153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6810","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6811","nonce":718927153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6814","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6815","nonce":718927153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6812","nonce":718927153},{"type":"v1","addr":"172.21.15.170:6813","nonce":718927153}]},"public_addr":"172.21.15.170:6809/718927153","cluster_addr":"172.21.15.170:6811/718927153","heartbeat_back_addr":"172.21.15.170:6815/718927153","heartbeat_front_addr":"172.21.15.170:6813/718927153","state":["exists","up"]},{"osd":6,"uuid":"25f7a3a4-3f5f-4625-8f87-766305b481c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6819","nonce":757778059}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6821","nonce":757778059}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6825","nonce":757778059}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":757778059},{"type":"v1","addr":"172.21.15.38:6823","nonce":757778059}]},"public_addr":"172.21.15.38:6819/757778059","cluster_addr":"172.21.15.38:6821/757778059","heartbeat_back_addr":"172.21.15.38:6825/757778059","heartbeat_front_addr":"172.21.15.38:6823/757778059","state":["exists","up"]},{"osd":7,"uuid":"11318d82-7c42-4e41-81b9-ecf84b65cd61","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6817","nonce":446835891}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6819","nonce":446835891}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6823","nonce":446835891}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":446835891},{"type":"v1","addr":"172.21.15.121:6821","nonce":446835891}]},"public_addr":"172.21.15.121:6817/446835891","cluster_addr":"172.21.15.121:6819/446835891","heartbeat_back_addr":"172.21.15.121:6823/446835891","heartbeat_front_addr":"172.21.15.121:6821/446835891","state":["exists","up"]},{"osd":8,"uuid":"fa713a9f-99ae-4b79-aa25-79dc9cf6b2f0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6816","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6817","nonce":1764725971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6818","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6819","nonce":1764725971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6822","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6823","nonce":1764725971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6820","nonce":1764725971},{"type":"v1","addr":"172.21.15.170:6821","nonce":1764725971}]},"public_addr":"172.21.15.170:6817/1764725971","cluster_addr":"172.21.15.170:6819/1764725971","heartbeat_back_addr":"172.21.15.170:6823/1764725971","heartbeat_front_addr":"172.21.15.170:6821/1764725971","state":["exists","up"]},{"osd":9,"uuid":"0fa2735f-8c3c-41bd-8de1-2681f659088c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6827","nonce":539511510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6829","nonce":539511510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6833","nonce":539511510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":539511510},{"type":"v1","addr":"172.21.15.38:6831","nonce":539511510}]},"public_addr":"172.21.15.38:6827/539511510","cluster_addr":"172.21.15.38:6829/539511510","heartbeat_back_addr":"172.21.15.38:6833/539511510","heartbeat_front_addr":"172.21.15.38:6831/539511510","state":["exists","up"]},{"osd":10,"uuid":"fa25661a-1c05-40d1-a17c-94d274eb01f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6825","nonce":1310211853}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6827","nonce":1310211853}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6831","nonce":1310211853}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":1310211853},{"type":"v1","addr":"172.21.15.121:6829","nonce":1310211853}]},"public_addr":"172.21.15.121:6825/1310211853","cluster_addr":"172.21.15.121:6827/1310211853","heartbeat_back_addr":"172.21.15.121:6831/1310211853","heartbeat_front_addr":"172.21.15.121:6829/1310211853","state":["exists","up"]},{"osd":11,"uuid":"c870b245-cb9a-4275-9b4f-6f86c049991a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6824","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6825","nonce":3915107754}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6826","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6827","nonce":3915107754}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6830","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6831","nonce":3915107754}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.170:6828","nonce":3915107754},{"type":"v1","addr":"172.21.15.170:6829","nonce":3915107754}]},"public_addr":"172.21.15.170:6825/3915107754","cluster_addr":"172.21.15.170:6827/3915107754","heartbeat_back_addr":"172.21.15.170:6831/3915107754","heartbeat_front_addr":"172.21.15.170:6829/3915107754","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:00.632651+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:24.362102+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:30:48.119684+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:14.430192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:31:38.523659+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:02.314002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:28.409201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:32:53.161532+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:17.640469+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:33:44.144132+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:09.184003+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T06:34:33.986973+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":7,"to":11}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.2d","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/3896600378":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/3929636377":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2512207794":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/2579773966":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/1523129198":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6801/2716021457":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/2750459261":"2024-03-27T06:37:32.618701+0000","172.21.15.38:6801/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6800/4246298230":"2024-03-27T06:28:26.137775+0000","172.21.15.38:0/2527804067":"2024-03-27T06:28:26.137775+0000","172.21.15.38:6800/1985135432":"2024-03-27T06:27:59.817781+0000","172.21.15.38:0/1675065388":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/4027757314":"2024-03-27T06:37:32.618701+0000","172.21.15.38:6800/2716021457":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/3320689790":"2024-03-27T06:37:32.618701+0000","172.21.15.38:0/3570258347":"2024-03-27T06:37:32.618701+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T06:48:16.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.024+0000 7f5b79553640 1 -- 172.21.15.38:0/1825104563 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f5b5c0a29f0 msgr2=0x7f5b5c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:16.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.024+0000 7f5b79553640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f5b5c0a29f0 0x7f5b5c0a4eb0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5b74198ac0 tx=0x7f5b6400b390 comp rx=0 tx=0).stop 2024-03-26T06:48:16.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.024+0000 7f5b79553640 1 -- 172.21.15.38:0/1825104563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74197980 msgr2=0x7f5b74197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:16.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.024+0000 7f5b79553640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74197980 0x7f5b74197e30 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c02c700 tx=0x7f5b6c032f20 comp rx=0 tx=0).stop 2024-03-26T06:48:16.028 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.025+0000 7f5b79553640 1 -- 172.21.15.38:0/1825104563 shutdown_connections 2024-03-26T06:48:16.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.025+0000 7f5b79553640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f5b5c0a29f0 0x7f5b5c0a4eb0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.025+0000 7f5b79553640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b74197980 0x7f5b74197e30 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.025+0000 7f5b79553640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f5b74106610 0x7f5b7419dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.025+0000 7f5b79553640 1 --2- 172.21.15.38:0/1825104563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5b74069c10 0x7f5b7419d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.025+0000 7f5b79553640 1 -- 172.21.15.38:0/1825104563 >> 172.21.15.38:0/1825104563 conn(0x7f5b74077290 msgr2=0x7f5b740fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:16.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.025+0000 7f5b79553640 1 -- 172.21.15.38:0/1825104563 shutdown_connections 2024-03-26T06:48:16.029 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.025+0000 7f5b79553640 1 -- 172.21.15.38:0/1825104563 wait complete. 2024-03-26T06:48:16.044 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-26T06:48:16.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.146+0000 7f011057e640 1 -- 172.21.15.38:0/1625430037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0108068760 msgr2=0x7f0108068b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:16.151 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.146+0000 7f011057e640 1 --2- 172.21.15.38:0/1625430037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0108068760 0x7f0108068b40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0104009900 tx=0x7f010402bc30 comp rx=0 tx=0).stop 2024-03-26T06:48:16.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.146+0000 7f011057e640 1 -- 172.21.15.38:0/1625430037 shutdown_connections 2024-03-26T06:48:16.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.146+0000 7f011057e640 1 --2- 172.21.15.38:0/1625430037 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0108106630 0x7f010810e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.146+0000 7f011057e640 1 --2- 172.21.15.38:0/1625430037 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0108069110 0x7f0108105f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.146+0000 7f011057e640 1 --2- 172.21.15.38:0/1625430037 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0108068760 0x7f0108068b40 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.146+0000 7f011057e640 1 -- 172.21.15.38:0/1625430037 >> 172.21.15.38:0/1625430037 conn(0x7f01080772f0 msgr2=0x7f0108077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:16.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.146+0000 7f011057e640 1 -- 172.21.15.38:0/1625430037 shutdown_connections 2024-03-26T06:48:16.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.146+0000 7f011057e640 1 -- 172.21.15.38:0/1625430037 wait complete. 2024-03-26T06:48:16.152 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.146+0000 7f011057e640 1 Processor -- start 2024-03-26T06:48:16.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f011057e640 1 -- start start 2024-03-26T06:48:16.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f011057e640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0108068760 0x7f010819d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:16.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f011057e640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0108106630 0x7f010819dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:16.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f011057e640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0108197920 0x7f0108197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:16.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f011057e640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0108111660 con 0x7f0108106630 2024-03-26T06:48:16.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f011057e640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01081114e0 con 0x7f0108197920 2024-03-26T06:48:16.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f011057e640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01081117e0 con 0x7f0108068760 2024-03-26T06:48:16.153 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f010db19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0108106630 0x7f010819dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:16.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f010db19640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0108106630 0x7f010819dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47164/0 (socket says 172.21.15.38:47164) 2024-03-26T06:48:16.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f010db19640 1 -- 172.21.15.38:0/2795291722 learned_addr learned my addr 172.21.15.38:0/2795291722 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:16.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f010eb1b640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0108197920 0x7f0108197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:16.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f010e31a640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0108068760 0x7f010819d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:16.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f010db19640 1 -- 172.21.15.38:0/2795291722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0108068760 msgr2=0x7f010819d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:16.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f010db19640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0108068760 0x7f010819d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f010db19640 1 -- 172.21.15.38:0/2795291722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0108197920 msgr2=0x7f0108197dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:16.154 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f010db19640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0108197920 0x7f0108197dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f010db19640 1 -- 172.21.15.38:0/2795291722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0108198440 con 0x7f0108106630 2024-03-26T06:48:16.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f010db19640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0108106630 0x7f010819dcb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f00f800d950 tx=0x7f00f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:16.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.147+0000 7f00ff7fe640 1 -- 172.21.15.38:0/2795291722 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f8017070 con 0x7f0108106630 2024-03-26T06:48:16.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.148+0000 7f011057e640 1 -- 172.21.15.38:0/2795291722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0108198730 con 0x7f0108106630 2024-03-26T06:48:16.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.148+0000 7f00ff7fe640 1 -- 172.21.15.38:0/2795291722 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00f8009c30 con 0x7f0108106630 2024-03-26T06:48:16.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.148+0000 7f00ff7fe640 1 -- 172.21.15.38:0/2795291722 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f80047a0 con 0x7f0108106630 2024-03-26T06:48:16.155 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.148+0000 7f011057e640 1 -- 172.21.15.38:0/2795291722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01081a49d0 con 0x7f0108106630 2024-03-26T06:48:16.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.149+0000 7f011057e640 1 -- 172.21.15.38:0/2795291722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01080680e0 con 0x7f0108106630 2024-03-26T06:48:16.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.150+0000 7f00ff7fe640 1 -- 172.21.15.38:0/2795291722 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f00f8004020 con 0x7f0108106630 2024-03-26T06:48:16.156 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.151+0000 7f00ff7fe640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f00f40a2970 0x7f00f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:16.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.151+0000 7f010e31a640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f00f40a2970 0x7f00f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:16.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.151+0000 7f00ff7fe640 1 -- 172.21.15.38:0/2795291722 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8629+0+0 (secure 0 0 0) 0x7f00f80ceb20 con 0x7f0108106630 2024-03-26T06:48:16.157 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.152+0000 7f010e31a640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f00f40a2970 0x7f00f40a4e30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0104009230 tx=0x7f0104005fd0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:16.159 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.156+0000 7f00ff7fe640 1 -- 172.21.15.38:0/2795291722 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f00f8020020 con 0x7f0108106630 2024-03-26T06:48:16.383 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.380+0000 7f011057e640 1 -- 172.21.15.38:0/2795291722 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0108109820 con 0x7f0108106630 2024-03-26T06:48:16.389 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.386+0000 7f00ff7fe640 1 -- 172.21.15.38:0/2795291722 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v31) v1 ==== 161+0+0 (secure 0 0 0) 0x7f00f809b810 con 0x7f0108106630 2024-03-26T06:48:16.390 INFO:teuthology.orchestra.run.smithi038.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-26T06:48:16.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 -- 172.21.15.38:0/2795291722 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f00f40a2970 msgr2=0x7f00f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:16.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f00f40a2970 0x7f00f40a4e30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0104009230 tx=0x7f0104005fd0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 -- 172.21.15.38:0/2795291722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0108106630 msgr2=0x7f010819dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:16.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0108106630 0x7f010819dcb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f00f800d950 tx=0x7f00f800de10 comp rx=0 tx=0).stop 2024-03-26T06:48:16.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 -- 172.21.15.38:0/2795291722 shutdown_connections 2024-03-26T06:48:16.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f00f40a2970 0x7f00f40a4e30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.393 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0108197920 0x7f0108197dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0108106630 0x7f010819dcb0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 --2- 172.21.15.38:0/2795291722 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f0108068760 0x7f010819d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 -- 172.21.15.38:0/2795291722 >> 172.21.15.38:0/2795291722 conn(0x7f01080772f0 msgr2=0x7f01080fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:16.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 -- 172.21.15.38:0/2795291722 shutdown_connections 2024-03-26T06:48:16.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.389+0000 7f011057e640 1 -- 172.21.15.38:0/2795291722 wait complete. 2024-03-26T06:48:16.406 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-26T06:48:16.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 -- 172.21.15.38:0/860153701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100460 msgr2=0x7f7710100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:16.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 --2- 172.21.15.38:0/860153701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100460 0x7f7710100840 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f770c009860 tx=0x7f770c02bb70 comp rx=0 tx=0).stop 2024-03-26T06:48:16.518 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 -- 172.21.15.38:0/860153701 shutdown_connections 2024-03-26T06:48:16.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 --2- 172.21.15.38:0/860153701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77101054e0 0x7f771010d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 --2- 172.21.15.38:0/860153701 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7710100e10 0x7f7710104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 --2- 172.21.15.38:0/860153701 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100460 0x7f7710100840 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 -- 172.21.15.38:0/860153701 >> 172.21.15.38:0/860153701 conn(0x7f7710077350 msgr2=0x7f7710077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:16.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 -- 172.21.15.38:0/860153701 shutdown_connections 2024-03-26T06:48:16.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 -- 172.21.15.38:0/860153701 wait complete. 2024-03-26T06:48:16.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 Processor -- start 2024-03-26T06:48:16.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 -- start start 2024-03-26T06:48:16.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7710100460 0x7f771019d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:16.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77101054e0 0x7f771019dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:16.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710197910 0x7f7710197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:16.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7710110570 con 0x7f7710197910 2024-03-26T06:48:16.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77101103f0 con 0x7f77101054e0 2024-03-26T06:48:16.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.513+0000 7f7717d53640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77101106f0 con 0x7f7710100460 2024-03-26T06:48:16.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f77162f0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710197910 0x7f7710197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:16.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f77162f0640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710197910 0x7f7710197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47180/0 (socket says 172.21.15.38:47180) 2024-03-26T06:48:16.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f77162f0640 1 -- 172.21.15.38:0/3491271443 learned_addr learned my addr 172.21.15.38:0/3491271443 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:16.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f77152ee640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77101054e0 0x7f771019dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:16.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f7715aef640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7710100460 0x7f771019d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:16.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f77162f0640 1 -- 172.21.15.38:0/3491271443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7710100460 msgr2=0x7f771019d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:16.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f77162f0640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7710100460 0x7f771019d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f77162f0640 1 -- 172.21.15.38:0/3491271443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77101054e0 msgr2=0x7f771019dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:16.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f77162f0640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77101054e0 0x7f771019dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:16.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f77162f0640 1 -- 172.21.15.38:0/3491271443 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7710198430 con 0x7f7710197910 2024-03-26T06:48:16.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f77162f0640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710197910 0x7f7710197dc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f770800ef00 tx=0x7f770800c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:16.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f7706ffd640 1 -- 172.21.15.38:0/3491271443 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7708003970 con 0x7f7710197910 2024-03-26T06:48:16.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f7706ffd640 1 -- 172.21.15.38:0/3491271443 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7708004020 con 0x7f7710197910 2024-03-26T06:48:16.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f7717d53640 1 -- 172.21.15.38:0/3491271443 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7710198720 con 0x7f7710197910 2024-03-26T06:48:16.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f7706ffd640 1 -- 172.21.15.38:0/3491271443 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770800b930 con 0x7f7710197910 2024-03-26T06:48:16.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.514+0000 7f7717d53640 1 -- 172.21.15.38:0/3491271443 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77101a4920 con 0x7f7710197910 2024-03-26T06:48:16.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.515+0000 7f7717d53640 1 -- 172.21.15.38:0/3491271443 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7710067850 con 0x7f7710197910 2024-03-26T06:48:16.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.517+0000 7f7706ffd640 1 -- 172.21.15.38:0/3491271443 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f770801d020 con 0x7f7710197910 2024-03-26T06:48:16.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.518+0000 7f7706ffd640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f76fc0a2970 0x7f76fc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:16.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.518+0000 7f7715aef640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f76fc0a2970 0x7f76fc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:16.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.518+0000 7f7706ffd640 1 -- 172.21.15.38:0/3491271443 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8843+0+0 (secure 0 0 0) 0x7f77080cedb0 con 0x7f7710197910 2024-03-26T06:48:16.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.518+0000 7f7715aef640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f76fc0a2970 0x7f76fc0a4e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f770c02c180 tx=0x7f770c0058b0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:16.526 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.523+0000 7f7706ffd640 1 -- 172.21.15.38:0/3491271443 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f77080d3070 con 0x7f7710197910 2024-03-26T06:48:16.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:16 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a[32261]: 2024-03-26T06:48:16.381+0000 7f6484b16700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T06:48:16.751 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:16.748+0000 7f7717d53640 1 -- 172.21.15.38:0/3491271443 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7710100e10 con 0x7f7710197910 2024-03-26T06:48:16.797 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b[66818]: ignoring --setuser ceph since I am not root 2024-03-26T06:48:16.797 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b[66818]: ignoring --setgroup ceph since I am not root 2024-03-26T06:48:16.797 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mds[66822]: -- 172.21.15.121:0/2603236751 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5630bd22da40 con 0x5630bd200800 2024-03-26T06:48:16.797 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b[66818]: starting mds.b at 2024-03-26T06:48:16.798 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h[66498]: ignoring --setuser ceph since I am not root 2024-03-26T06:48:16.798 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h[66498]: ignoring --setgroup ceph since I am not root 2024-03-26T06:48:16.798 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mds[66502]: -- 172.21.15.121:0/2542400876 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563315dc1a40 con 0x563315d95000 2024-03-26T06:48:16.798 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h[66498]: starting mds.h at 2024-03-26T06:48:16.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mon[36349]: pgmap v334: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-03-26T06:48:16.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1373749581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:48:16.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/1825104563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:48:16.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2795291722' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-26T06:48:16.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mon[36349]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T06:48:16.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mon[36349]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T06:48:16.799 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2795291722' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-26T06:48:16.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mon[36349]: osdmap e100: 12 total, 12 up, 12 in 2024-03-26T06:48:16.800 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mon[36349]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-26T06:48:16.800 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-e[66183]: ignoring --setuser ceph since I am not root 2024-03-26T06:48:16.800 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-e[66183]: ignoring --setgroup ceph since I am not root 2024-03-26T06:48:16.800 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-mds[66187]: -- 172.21.15.121:0/334010589 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559c458ada40 con 0x559c45881000 2024-03-26T06:48:16.800 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:48:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-e[66183]: starting mds.e at 2024-03-26T06:48:16.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mon[36912]: pgmap v334: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-03-26T06:48:16.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1373749581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:48:16.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/1825104563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:48:16.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2795291722' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-26T06:48:16.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mon[36912]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T06:48:16.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mon[36912]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T06:48:16.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2795291722' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-26T06:48:16.921 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mon[36912]: osdmap e100: 12 total, 12 up, 12 in 2024-03-26T06:48:16.921 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mon[36912]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-26T06:48:16.921 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f[67728]: ignoring --setuser ceph since I am not root 2024-03-26T06:48:16.922 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f[67728]: ignoring --setgroup ceph since I am not root 2024-03-26T06:48:16.922 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mds[67732]: -- 172.21.15.170:0/2994969526 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561fdba3fa40 con 0x561fdba12c00 2024-03-26T06:48:16.922 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f[67728]: starting mds.f at 2024-03-26T06:48:16.923 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i[68353]: ignoring --setuser ceph since I am not root 2024-03-26T06:48:16.923 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i[68353]: ignoring --setgroup ceph since I am not root 2024-03-26T06:48:16.924 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mds[68357]: -- 172.21.15.170:0/2748252246 <== mon.2 v2:172.21.15.121:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dc50e7ba40 con 0x55dc50e4e800 2024-03-26T06:48:16.924 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i[68353]: starting mds.i at 2024-03-26T06:48:16.924 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[71439]: ignoring --setuser ceph since I am not root 2024-03-26T06:48:16.925 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[71439]: ignoring --setgroup ceph since I am not root 2024-03-26T06:48:16.925 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-mds[71443]: -- 172.21.15.170:0/2266813640 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d8eace5a40 con 0x55d8eacb8800 2024-03-26T06:48:16.925 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:48:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[71439]: starting mds.c at 2024-03-26T06:48:17.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:16 smithi038 ceph-mon[32265]: pgmap v334: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 14 KiB/s rd, 26 KiB/s wr, 23 op/s 2024-03-26T06:48:17.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:16 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1373749581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-26T06:48:17.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:16 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/1825104563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T06:48:17.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:16 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2795291722' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-26T06:48:17.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:16 smithi038 ceph-mon[32265]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-26T06:48:17.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:16 smithi038 ceph-mon[32265]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-26T06:48:17.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:16 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2795291722' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-26T06:48:17.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:16 smithi038 ceph-mon[32265]: osdmap e100: 12 total, 12 up, 12 in 2024-03-26T06:48:17.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:16 smithi038 ceph-mon[32265]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-26T06:48:17.391 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.388+0000 7f7706ffd640 1 -- 172.21.15.38:0/3491271443 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v32) v1 ==== 104+0+0 (secure 0 0 0) 0x7f770809b970 con 0x7f7710197910 2024-03-26T06:48:17.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.390+0000 7f7717d53640 1 -- 172.21.15.38:0/3491271443 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f76fc0a2970 msgr2=0x7f76fc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:17.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.390+0000 7f7717d53640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f76fc0a2970 0x7f76fc0a4e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f770c02c180 tx=0x7f770c0058b0 comp rx=0 tx=0).stop 2024-03-26T06:48:17.394 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.390+0000 7f7717d53640 1 -- 172.21.15.38:0/3491271443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710197910 msgr2=0x7f7710197dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:17.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.390+0000 7f7717d53640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710197910 0x7f7710197dc0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f770800ef00 tx=0x7f770800c740 comp rx=0 tx=0).stop 2024-03-26T06:48:17.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.391+0000 7f7717d53640 1 -- 172.21.15.38:0/3491271443 shutdown_connections 2024-03-26T06:48:17.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.391+0000 7f7717d53640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f76fc0a2970 0x7f76fc0a4e30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:17.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.391+0000 7f7717d53640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710197910 0x7f7710197dc0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:17.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.391+0000 7f7717d53640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77101054e0 0x7f771019dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:17.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.391+0000 7f7717d53640 1 --2- 172.21.15.38:0/3491271443 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f7710100460 0x7f771019d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:17.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.391+0000 7f7717d53640 1 -- 172.21.15.38:0/3491271443 >> 172.21.15.38:0/3491271443 conn(0x7f7710077350 msgr2=0x7f7710069830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:17.395 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.391+0000 7f7717d53640 1 -- 172.21.15.38:0/3491271443 shutdown_connections 2024-03-26T06:48:17.396 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.391+0000 7f7717d53640 1 -- 172.21.15.38:0/3491271443 wait complete. 2024-03-26T06:48:17.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-26T06:48:17.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.513+0000 7fa8cf244640 1 -- 172.21.15.38:0/641430275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8c8100dc0 msgr2=0x7fa8c8104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:17.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.513+0000 7fa8cf244640 1 --2- 172.21.15.38:0/641430275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8c8100dc0 0x7fa8c8104e00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4009900 tx=0x7fa8b402bc30 comp rx=0 tx=0).stop 2024-03-26T06:48:17.519 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.513+0000 7fa8cf244640 1 -- 172.21.15.38:0/641430275 shutdown_connections 2024-03-26T06:48:17.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.513+0000 7fa8cf244640 1 --2- 172.21.15.38:0/641430275 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c81054e0 0x7fa8c810d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:17.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.513+0000 7fa8cf244640 1 --2- 172.21.15.38:0/641430275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8c8100dc0 0x7fa8c8104e00 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:17.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.513+0000 7fa8cf244640 1 --2- 172.21.15.38:0/641430275 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c8100410 0x7fa8c81007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:17.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.513+0000 7fa8cf244640 1 -- 172.21.15.38:0/641430275 >> 172.21.15.38:0/641430275 conn(0x7fa8c8076570 msgr2=0x7fa8c8076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:17.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.513+0000 7fa8cf244640 1 -- 172.21.15.38:0/641430275 shutdown_connections 2024-03-26T06:48:17.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.513+0000 7fa8cf244640 1 -- 172.21.15.38:0/641430275 wait complete. 2024-03-26T06:48:17.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8cf244640 1 Processor -- start 2024-03-26T06:48:17.520 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8cf244640 1 -- start start 2024-03-26T06:48:17.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8cf244640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c8100410 0x7fa8c819d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:17.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8cf244640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8c81054e0 0x7fa8c819dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:17.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8cf244640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c8197950 0x7fa8c8197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:17.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8cf244640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c8110550 con 0x7fa8c81054e0 2024-03-26T06:48:17.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8cf244640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c81103d0 con 0x7fa8c8100410 2024-03-26T06:48:17.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8cf244640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c81106d0 con 0x7fa8c8197950 2024-03-26T06:48:17.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8c7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8c81054e0 0x7fa8c819dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:17.521 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8c7fff640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8c81054e0 0x7fa8c819dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47192/0 (socket says 172.21.15.38:47192) 2024-03-26T06:48:17.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8ccfe0640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c8100410 0x7fa8c819d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:17.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8c7fff640 1 -- 172.21.15.38:0/2305091415 learned_addr learned my addr 172.21.15.38:0/2305091415 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:17.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8cd7e1640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c8197950 0x7fa8c8197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:17.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8c7fff640 1 -- 172.21.15.38:0/2305091415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c8197950 msgr2=0x7fa8c8197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:17.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8c7fff640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c8197950 0x7fa8c8197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:17.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8c7fff640 1 -- 172.21.15.38:0/2305091415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c8100410 msgr2=0x7fa8c819d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:17.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8c7fff640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c8100410 0x7fa8c819d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:17.522 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8c7fff640 1 -- 172.21.15.38:0/2305091415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c8198470 con 0x7fa8c81054e0 2024-03-26T06:48:17.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8ccfe0640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c8100410 0x7fa8c819d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T06:48:17.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.514+0000 7fa8cd7e1640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c8197950 0x7fa8c8197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:48:17.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.515+0000 7fa8c7fff640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8c81054e0 0x7fa8c819dce0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4036040 tx=0x7fa8b402ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:17.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.515+0000 7fa8c5ffb640 1 -- 172.21.15.38:0/2305091415 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4041070 con 0x7fa8c81054e0 2024-03-26T06:48:17.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.515+0000 7fa8c5ffb640 1 -- 172.21.15.38:0/2305091415 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8b40054e0 con 0x7fa8c81054e0 2024-03-26T06:48:17.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.515+0000 7fa8cf244640 1 -- 172.21.15.38:0/2305091415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c8198700 con 0x7fa8c81054e0 2024-03-26T06:48:17.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.515+0000 7fa8c5ffb640 1 -- 172.21.15.38:0/2305091415 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4034440 con 0x7fa8c81054e0 2024-03-26T06:48:17.523 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.515+0000 7fa8cf244640 1 -- 172.21.15.38:0/2305091415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c81a48f0 con 0x7fa8c81054e0 2024-03-26T06:48:17.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.516+0000 7fa8cf244640 1 -- 172.21.15.38:0/2305091415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c81018e0 con 0x7fa8c81054e0 2024-03-26T06:48:17.524 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.517+0000 7fa8c5ffb640 1 -- 172.21.15.38:0/2305091415 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa8b4044070 con 0x7fa8c81054e0 2024-03-26T06:48:17.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.517+0000 7fa8c5ffb640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa8b80a2970 0x7fa8b80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:17.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.518+0000 7fa8ccfe0640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa8b80a2970 0x7fa8b80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:17.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.518+0000 7fa8c5ffb640 1 -- 172.21.15.38:0/2305091415 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 8843+0+0 (secure 0 0 0) 0x7fa8b40f0b20 con 0x7fa8c81054e0 2024-03-26T06:48:17.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.518+0000 7fa8ccfe0640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa8b80a2970 0x7fa8b80a4e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8198a90 tx=0x7fa8bc00b410 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:17.525 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.522+0000 7fa8c5ffb640 1 -- 172.21.15.38:0/2305091415 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa8b40bd8e0 con 0x7fa8c81054e0 2024-03-26T06:48:17.708 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:17.705+0000 7fa8cf244640 1 -- 172.21.15.38:0/2305091415 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa8c8100dc0 con 0x7fa8c81054e0 2024-03-26T06:48:17.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3491271443' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T06:48:17.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T06:48:17.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T06:48:17.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: Cluster is now healthy 2024-03-26T06:48:17.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/3491271443' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-26T06:48:17.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.170:6834/742766065,v1:172.21.15.170:6837/742766065] up:boot 2024-03-26T06:48:17.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.121:6838/930404314,v1:172.21.15.121:6839/930404314] up:boot 2024-03-26T06:48:17.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.121:6834/3227370505,v1:172.21.15.121:6835/3227370505] up:boot 2024-03-26T06:48:17.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.170:6836/2689255487,v1:172.21.15.170:6838/2689255487] up:boot 2024-03-26T06:48:17.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.170:6835/1599424026,v1:172.21.15.170:6839/1599424026] up:boot 2024-03-26T06:48:17.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: mds.? [v2:172.21.15.121:6836/2591340990,v1:172.21.15.121:6837/2591340990] up:boot 2024-03-26T06:48:17.897 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: fsmap 12 up:standby 2024-03-26T06:48:17.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-26T06:48:17.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-26T06:48:17.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:48:17.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-26T06:48:17.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-26T06:48:17.898 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:17 smithi121 ceph-mon[36349]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-26T06:48:17.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3491271443' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T06:48:17.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T06:48:17.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T06:48:17.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: Cluster is now healthy 2024-03-26T06:48:17.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/3491271443' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-26T06:48:17.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.170:6834/742766065,v1:172.21.15.170:6837/742766065] up:boot 2024-03-26T06:48:17.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.121:6838/930404314,v1:172.21.15.121:6839/930404314] up:boot 2024-03-26T06:48:17.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.121:6834/3227370505,v1:172.21.15.121:6835/3227370505] up:boot 2024-03-26T06:48:17.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.170:6836/2689255487,v1:172.21.15.170:6838/2689255487] up:boot 2024-03-26T06:48:17.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.170:6835/1599424026,v1:172.21.15.170:6839/1599424026] up:boot 2024-03-26T06:48:17.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: mds.? [v2:172.21.15.121:6836/2591340990,v1:172.21.15.121:6837/2591340990] up:boot 2024-03-26T06:48:17.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: fsmap 12 up:standby 2024-03-26T06:48:17.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-26T06:48:17.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-26T06:48:17.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:48:17.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-26T06:48:17.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-26T06:48:17.920 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:17 smithi170 ceph-mon[36912]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-26T06:48:18.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3491271443' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T06:48:18.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-26T06:48:18.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-26T06:48:18.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: Cluster is now healthy 2024-03-26T06:48:18.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/3491271443' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-26T06:48:18.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.170:6834/742766065,v1:172.21.15.170:6837/742766065] up:boot 2024-03-26T06:48:18.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.121:6838/930404314,v1:172.21.15.121:6839/930404314] up:boot 2024-03-26T06:48:18.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.121:6834/3227370505,v1:172.21.15.121:6835/3227370505] up:boot 2024-03-26T06:48:18.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.170:6836/2689255487,v1:172.21.15.170:6838/2689255487] up:boot 2024-03-26T06:48:18.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.170:6835/1599424026,v1:172.21.15.170:6839/1599424026] up:boot 2024-03-26T06:48:18.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: mds.? [v2:172.21.15.121:6836/2591340990,v1:172.21.15.121:6837/2591340990] up:boot 2024-03-26T06:48:18.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: fsmap 12 up:standby 2024-03-26T06:48:18.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-26T06:48:18.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-26T06:48:18.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-26T06:48:18.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-26T06:48:18.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-26T06:48:18.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:17 smithi038 ceph-mon[32265]: from='mgr.24883 172.21.15.38:0/2793662850' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-26T06:48:18.094 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.091+0000 7fa8c5ffb640 1 -- 172.21.15.38:0/2305091415 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa8b4004c50 con 0x7fa8c81054e0 2024-03-26T06:48:18.094 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_metadata' removed 2024-03-26T06:48:18.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.096+0000 7fa8cf244640 1 -- 172.21.15.38:0/2305091415 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa8b80a2970 msgr2=0x7fa8b80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:18.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.096+0000 7fa8cf244640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa8b80a2970 0x7fa8b80a4e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8198a90 tx=0x7fa8bc00b410 comp rx=0 tx=0).stop 2024-03-26T06:48:18.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.096+0000 7fa8cf244640 1 -- 172.21.15.38:0/2305091415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8c81054e0 msgr2=0x7fa8c819dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:18.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.096+0000 7fa8cf244640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8c81054e0 0x7fa8c819dce0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fa8b4036040 tx=0x7fa8b402ce20 comp rx=0 tx=0).stop 2024-03-26T06:48:18.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.096+0000 7fa8cf244640 1 -- 172.21.15.38:0/2305091415 shutdown_connections 2024-03-26T06:48:18.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.096+0000 7fa8cf244640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7fa8b80a2970 0x7fa8b80a4e30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:18.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.096+0000 7fa8cf244640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7fa8c8197950 0x7fa8c8197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:18.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.096+0000 7fa8cf244640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa8c81054e0 0x7fa8c819dce0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:18.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.096+0000 7fa8cf244640 1 --2- 172.21.15.38:0/2305091415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa8c8100410 0x7fa8c819d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:18.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.096+0000 7fa8cf244640 1 -- 172.21.15.38:0/2305091415 >> 172.21.15.38:0/2305091415 conn(0x7fa8c8076570 msgr2=0x7fa8c8069ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:18.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.097+0000 7fa8cf244640 1 -- 172.21.15.38:0/2305091415 shutdown_connections 2024-03-26T06:48:18.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.097+0000 7fa8cf244640 1 -- 172.21.15.38:0/2305091415 wait complete. 2024-03-26T06:48:18.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-26T06:48:18.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.220+0000 7f75cbb69640 1 -- 172.21.15.38:0/2214927347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c4101e80 msgr2=0x7f75c4105ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:18.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.220+0000 7f75cbb69640 1 --2- 172.21.15.38:0/2214927347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c4101e80 0x7f75c4105ec0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009900 tx=0x7f75b402bc30 comp rx=0 tx=0).stop 2024-03-26T06:48:18.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.220+0000 7f75cbb69640 1 -- 172.21.15.38:0/2214927347 shutdown_connections 2024-03-26T06:48:18.226 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.220+0000 7f75cbb69640 1 --2- 172.21.15.38:0/2214927347 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75c41065a0 0x7f75c410e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:18.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.220+0000 7f75cbb69640 1 --2- 172.21.15.38:0/2214927347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c4101e80 0x7f75c4105ec0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:18.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.220+0000 7f75cbb69640 1 --2- 172.21.15.38:0/2214927347 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c41014d0 0x7f75c41018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:18.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.220+0000 7f75cbb69640 1 -- 172.21.15.38:0/2214927347 >> 172.21.15.38:0/2214927347 conn(0x7f75c40fcd90 msgr2=0x7f75c40ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:18.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.220+0000 7f75cbb69640 1 -- 172.21.15.38:0/2214927347 shutdown_connections 2024-03-26T06:48:18.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.220+0000 7f75cbb69640 1 -- 172.21.15.38:0/2214927347 wait complete. 2024-03-26T06:48:18.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75cbb69640 1 Processor -- start 2024-03-26T06:48:18.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75cbb69640 1 -- start start 2024-03-26T06:48:18.227 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75cbb69640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c41014d0 0x7f75c419d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:18.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75cbb69640 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75c41065a0 0x7f75c419dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:18.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75cbb69640 1 --2- >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4197a10 0x7f75c4197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:18.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75cbb69640 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c4111610 con 0x7f75c41014d0 2024-03-26T06:48:18.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75cbb69640 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c4111490 con 0x7f75c41065a0 2024-03-26T06:48:18.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75cbb69640 1 -- --> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c4111790 con 0x7f75c4197a10 2024-03-26T06:48:18.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75c9905640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c41014d0 0x7f75c419d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:18.228 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75c9905640 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c41014d0 0x7f75c419d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:47212/0 (socket says 172.21.15.38:47212) 2024-03-26T06:48:18.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75c9905640 1 -- 172.21.15.38:0/2437164653 learned_addr learned my addr 172.21.15.38:0/2437164653 (peer_addr_for_me v2:172.21.15.38:0/0) 2024-03-26T06:48:18.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.221+0000 7f75c9104640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75c41065a0 0x7f75c419dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:18.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75ca106640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4197a10 0x7f75c4197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:18.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75c9905640 1 -- 172.21.15.38:0/2437164653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4197a10 msgr2=0x7f75c4197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:18.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75c9905640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4197a10 0x7f75c4197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:18.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75c9905640 1 -- 172.21.15.38:0/2437164653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75c41065a0 msgr2=0x7f75c419dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:18.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75c9905640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75c41065a0 0x7f75c419dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:18.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75c9905640 1 -- 172.21.15.38:0/2437164653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c4198530 con 0x7f75c41014d0 2024-03-26T06:48:18.229 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75ca106640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4197a10 0x7f75c4197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T06:48:18.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75c9905640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c41014d0 0x7f75c419d810 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f75c000c890 tx=0x7f75c000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:18.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75baffd640 1 -- 172.21.15.38:0/2437164653 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0004220 con 0x7f75c41014d0 2024-03-26T06:48:18.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75cbb69640 1 -- 172.21.15.38:0/2437164653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c4198820 con 0x7f75c41014d0 2024-03-26T06:48:18.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75baffd640 1 -- 172.21.15.38:0/2437164653 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75c00043c0 con 0x7f75c41014d0 2024-03-26T06:48:18.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75baffd640 1 -- 172.21.15.38:0/2437164653 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c00049e0 con 0x7f75c41014d0 2024-03-26T06:48:18.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.222+0000 7f75cbb69640 1 -- 172.21.15.38:0/2437164653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c41a4a20 con 0x7f75c41014d0 2024-03-26T06:48:18.230 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.223+0000 7f75cbb69640 1 -- 172.21.15.38:0/2437164653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c4068530 con 0x7f75c41014d0 2024-03-26T06:48:18.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.224+0000 7f75baffd640 1 -- 172.21.15.38:0/2437164653 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f75c00079b0 con 0x7f75c41014d0 2024-03-26T06:48:18.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.224+0000 7f75baffd640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f75b00a2970 0x7f75b00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T06:48:18.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.225+0000 7f75c9104640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f75b00a2970 0x7f75b00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T06:48:18.231 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.225+0000 7f75baffd640 1 -- 172.21.15.38:0/2437164653 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8289+0+0 (secure 0 0 0) 0x7f75c00cee50 con 0x7f75c41014d0 2024-03-26T06:48:18.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.225+0000 7f75c9104640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f75b00a2970 0x7f75b00a4e30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009230 tx=0x7f75b4005fd0 comp rx=0 tx=0).ready entity=mgr.24883 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T06:48:18.232 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.229+0000 7f75baffd640 1 -- 172.21.15.38:0/2437164653 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f75c0014030 con 0x7f75c41014d0 2024-03-26T06:48:18.420 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:18.417+0000 7f75cbb69640 1 -- 172.21.15.38:0/2437164653 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f75c4198ef0 con 0x7f75c41014d0 2024-03-26T06:48:18.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:18 smithi121 ceph-mon[36349]: pgmap v336: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 331 B/s wr, 5 op/s 2024-03-26T06:48:18.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:18 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2305091415' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T06:48:18.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:18 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2305091415' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T06:48:18.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:18 smithi121 ceph-mon[36349]: osdmap e101: 12 total, 12 up, 12 in 2024-03-26T06:48:18.896 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:18 smithi121 ceph-mon[36349]: from='client.? 172.21.15.38:0/2437164653' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T06:48:18.918 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:18 smithi170 ceph-mon[36912]: pgmap v336: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 331 B/s wr, 5 op/s 2024-03-26T06:48:18.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:18 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2305091415' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T06:48:18.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:18 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2305091415' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T06:48:18.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:18 smithi170 ceph-mon[36912]: osdmap e101: 12 total, 12 up, 12 in 2024-03-26T06:48:18.919 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:18 smithi170 ceph-mon[36912]: from='client.? 172.21.15.38:0/2437164653' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T06:48:19.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:18 smithi038 ceph-mon[32265]: pgmap v336: 97 pgs: 97 active+clean; 13 GiB data, 59 GiB used, 1014 GiB / 1.0 TiB avail; 2.9 KiB/s rd, 331 B/s wr, 5 op/s 2024-03-26T06:48:19.084 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:18 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2305091415' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T06:48:19.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:18 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2305091415' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T06:48:19.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:18 smithi038 ceph-mon[32265]: osdmap e101: 12 total, 12 up, 12 in 2024-03-26T06:48:19.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:18 smithi038 ceph-mon[32265]: from='client.? 172.21.15.38:0/2437164653' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T06:48:19.096 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.093+0000 7f75baffd640 1 -- 172.21.15.38:0/2437164653 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) v1 ==== 169+0+0 (secure 0 0 0) 0x7f75c009bc90 con 0x7f75c41014d0 2024-03-26T06:48:19.096 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data' removed 2024-03-26T06:48:19.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 -- 172.21.15.38:0/2437164653 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f75b00a2970 msgr2=0x7f75b00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:19.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f75b00a2970 0x7f75b00a4e30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f75b4009230 tx=0x7f75b4005fd0 comp rx=0 tx=0).stop 2024-03-26T06:48:19.099 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 -- 172.21.15.38:0/2437164653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c41014d0 msgr2=0x7f75c419d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T06:48:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c41014d0 0x7f75c419d810 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f75c000c890 tx=0x7f75c000cd50 comp rx=0 tx=0).stop 2024-03-26T06:48:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 -- 172.21.15.38:0/2437164653 shutdown_connections 2024-03-26T06:48:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.38:6800/1823987770,v1:172.21.15.38:6801/1823987770] conn(0x7f75b00a2970 0x7f75b00a4e30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.170:3300/0,v1:172.21.15.170:6789/0] conn(0x7f75c4197a10 0x7f75c4197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75c41065a0 0x7f75c419dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 --2- 172.21.15.38:0/2437164653 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f75c41014d0 0x7f75c419d810 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T06:48:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 -- 172.21.15.38:0/2437164653 >> 172.21.15.38:0/2437164653 conn(0x7f75c40fcd90 msgr2=0x7f75c40fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T06:48:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 -- 172.21.15.38:0/2437164653 shutdown_connections 2024-03-26T06:48:19.101 INFO:teuthology.orchestra.run.smithi038.stderr:2024-03-26T06:48:19.096+0000 7f75cbb69640 1 -- 172.21.15.38:0/2437164653 wait complete. 2024-03-26T06:48:19.121 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T06:48:19.173 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T06:48:19.226 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T06:48:19.280 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T06:48:19.281 DEBUG:teuthology.orchestra.run.smithi038:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T06:48:19.307 DEBUG:teuthology.orchestra.run.smithi121:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T06:48:19.333 DEBUG:teuthology.orchestra.run.smithi170:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T06:48:19.357 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T06:48:19.358 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T06:48:19.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.a 2024-03-26T06:48:19.834 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:19 smithi038 systemd[1]: Stopping Ceph mon.a for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:48:20.198 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a[32261]: 2024-03-26T06:48:19.869+0000 7f648a321700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:48:20.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a[32261]: 2024-03-26T06:48:19.869+0000 7f648a321700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T06:48:20.199 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:19 smithi038 podman[91242]: 2024-03-26 06:48:19.902360588 +0000 UTC m=+0.175697001 container died 127b2bec7e81d0af976eb8c76ed4bf74de9bdc731e4ed441271857d57627d0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T06:48:20.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:20 smithi038 podman[91242]: 2024-03-26 06:48:20.28629203 +0000 UTC m=+0.559628434 container cleanup 127b2bec7e81d0af976eb8c76ed4bf74de9bdc731e4ed441271857d57627d0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T06:48:20.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:20 smithi038 bash[91242]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a 2024-03-26T06:48:21.057 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 26 06:48:20 smithi038 podman[91252]: 2024-03-26 06:48:20.612863846 +0000 UTC m=+0.697292526 container remove 127b2bec7e81d0af976eb8c76ed4bf74de9bdc731e4ed441271857d57627d0bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319) 2024-03-26T06:48:21.072 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T06:48:21.169 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:48:21.169 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T06:48:21.169 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-26T06:48:21.170 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.b 2024-03-26T06:48:21.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:21 smithi121 systemd[1]: Stopping Ceph mon.b for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:48:22.147 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:21 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-b[36345]: 2024-03-26T06:48:21.808+0000 7f47e8721700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:48:22.356 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:21 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-b[36345]: 2024-03-26T06:48:21.808+0000 7f47e8721700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T06:48:22.357 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:21 smithi121 podman[72118]: 2024-03-26 06:48:21.93187591 +0000 UTC m=+0.309733910 container died 015bdab4d4367d90b5c051b53b142278167b30a657a85298be92af76897b7b54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:48:22.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:22 smithi121 podman[72118]: 2024-03-26 06:48:22.435634878 +0000 UTC m=+0.813492869 container cleanup 015bdab4d4367d90b5c051b53b142278167b30a657a85298be92af76897b7b54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-b, RELEASE=rishabh-2024mar8-2-7e98f5f, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T06:48:22.646 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:22 smithi121 bash[72118]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-b 2024-03-26T06:48:22.984 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T06:48:23.033 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:22 smithi121 podman[72128]: 2024-03-26 06:48:22.745423027 +0000 UTC m=+0.809221256 container remove 015bdab4d4367d90b5c051b53b142278167b30a657a85298be92af76897b7b54 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-b, RELEASE=rishabh-2024mar8-2-7e98f5f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-26T06:48:23.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:22 smithi121 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-26T06:48:23.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:22 smithi121 systemd[1]: Stopped Ceph mon.b for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:48:23.034 INFO:journalctl@ceph.mon.b.smithi121.stdout:Mar 26 06:48:22 smithi121 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.b.service: Consumed 12.523s CPU time. 2024-03-26T06:48:23.074 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:48:23.074 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-26T06:48:23.075 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-26T06:48:23.075 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.c 2024-03-26T06:48:23.418 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:23 smithi170 systemd[1]: Stopping Ceph mon.c for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:48:23.891 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:23 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-c[36906]: 2024-03-26T06:48:23.552+0000 7fe06f0ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:48:23.891 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:23 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-c[36906]: 2024-03-26T06:48:23.552+0000 7fe06f0ee700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T06:48:23.891 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:23 smithi170 podman[74095]: 2024-03-26 06:48:23.60511379 +0000 UTC m=+0.201258239 container died bceeb49f7778be465de549d43e5e625e348adf89283b41f1ba6c0277a7184fdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-c, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-26T06:48:24.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:23 smithi170 podman[74095]: 2024-03-26 06:48:23.968945629 +0000 UTC m=+0.565090083 container cleanup bceeb49f7778be465de549d43e5e625e348adf89283b41f1ba6c0277a7184fdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, RELEASE=rishabh-2024mar8-2-7e98f5f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-26T06:48:24.169 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:23 smithi170 bash[74095]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-c 2024-03-26T06:48:24.669 INFO:journalctl@ceph.mon.c.smithi170.stdout:Mar 26 06:48:24 smithi170 podman[74106]: 2024-03-26 06:48:24.287185944 +0000 UTC m=+0.681825287 container remove bceeb49f7778be465de549d43e5e625e348adf89283b41f1ba6c0277a7184fdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T06:48:24.797 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T06:48:24.877 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:48:24.877 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-26T06:48:24.877 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T06:48:24.878 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.x 2024-03-26T06:48:25.334 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:48:24 smithi038 systemd[1]: Stopping Ceph mgr.x for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:48:25.834 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:48:25 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x[32499]: 2024-03-26T06:48:25.386+0000 7f6513573700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:48:25.834 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:48:25 smithi038 podman[91345]: 2024-03-26 06:48:25.436112533 +0000 UTC m=+0.246506708 container died 1c39329e522c80af668165827c481e4b7550c34923c3c70b44bbf0108a908f6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, ceph=True) 2024-03-26T06:48:26.272 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:48:25 smithi038 podman[91345]: 2024-03-26 06:48:25.992319041 +0000 UTC m=+0.802713232 container cleanup 1c39329e522c80af668165827c481e4b7550c34923c3c70b44bbf0108a908f6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7e98f5f6b535defc7fa715b456904ffc0500aa9f, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:48:26.272 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 26 06:48:25 smithi038 bash[91345]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-x 2024-03-26T06:48:26.405 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T06:48:26.443 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:48:26.443 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T06:48:26.443 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T06:48:26.444 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.y 2024-03-26T06:48:26.789 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:48:26 smithi121 systemd[1]: Stopping Ceph mgr.y for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:48:27.134 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:48:26 smithi121 podman[72225]: 2024-03-26 06:48:26.78566792 +0000 UTC m=+0.159496847 container died bcb10b978ad01dc016e7efe882ff434a241b17bb9fac3334a8998fe0f8406b8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-26T06:48:27.396 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:48:27 smithi121 podman[72225]: 2024-03-26 06:48:27.131609713 +0000 UTC m=+0.505438650 container cleanup bcb10b978ad01dc016e7efe882ff434a241b17bb9fac3334a8998fe0f8406b8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-26T06:48:27.397 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:48:27 smithi121 bash[72225]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y 2024-03-26T06:48:27.397 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:48:27 smithi121 podman[72237]: 2024-03-26 06:48:27.30782761 +0000 UTC m=+0.518862842 container remove bcb10b978ad01dc016e7efe882ff434a241b17bb9fac3334a8998fe0f8406b8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2) 2024-03-26T06:48:27.397 INFO:journalctl@ceph.mgr.y.smithi121.stdout:Mar 26 06:48:27 smithi121 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-26T06:48:27.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T06:48:27.587 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:48:27.587 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T06:48:27.588 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-26T06:48:27.588 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.z 2024-03-26T06:48:27.918 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:48:27 smithi170 systemd[1]: Stopping Ceph mgr.z for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:48:28.211 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:48:27 smithi170 podman[74205]: 2024-03-26 06:48:27.944875156 +0000 UTC m=+0.139956600 container died 5062b4749cc800cef91d7edf1996ece6b171ff2309b42538a05496317c827453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f) 2024-03-26T06:48:28.577 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:48:28 smithi170 podman[74205]: 2024-03-26 06:48:28.307002565 +0000 UTC m=+0.502083994 container cleanup 5062b4749cc800cef91d7edf1996ece6b171ff2309b42538a05496317c827453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T06:48:28.577 INFO:journalctl@ceph.mgr.z.smithi170.stdout:Mar 26 06:48:28 smithi170 bash[74205]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mgr-z 2024-03-26T06:48:28.744 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mgr.z.service' 2024-03-26T06:48:28.786 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:48:28.786 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-26T06:48:28.786 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T06:48:28.787 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.0 2024-03-26T06:48:29.083 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:48:28 smithi038 systemd[1]: Stopping Ceph osd.0 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:48:29.334 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:48:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0[44309]: 2024-03-26T06:48:29.080+0000 7f911ba66700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:48:29.334 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:48:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0[44309]: 2024-03-26T06:48:29.080+0000 7f911ba66700 -1 osd.0 102 *** Got signal Terminated *** 2024-03-26T06:48:29.334 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:48:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0[44309]: 2024-03-26T06:48:29.080+0000 7f911ba66700 -1 osd.0 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:48:59.517 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:48:59 smithi038 bash[91461]: time="2024-03-26T06:48:59Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0 in 30 seconds, resorting to SIGKILL" 2024-03-26T06:48:59.518 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:48:59 smithi038 podman[91461]: 2024-03-26 06:48:59.235676215 +0000 UTC m=+30.273627822 container died 3711054aa6352c457d729d469d37f01fa0212f4e6e20988a5395f927894ac5c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, ceph=True) 2024-03-26T06:48:59.834 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:48:59 smithi038 podman[91461]: 2024-03-26 06:48:59.613733538 +0000 UTC m=+30.651685143 container cleanup 3711054aa6352c457d729d469d37f01fa0212f4e6e20988a5395f927894ac5c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T06:48:59.834 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:48:59 smithi038 bash[91461]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0 2024-03-26T06:49:00.334 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:48:59 smithi038 podman[91495]: 2024-03-26 06:48:59.998818281 +0000 UTC m=+0.761344597 container remove 3711054aa6352c457d729d469d37f01fa0212f4e6e20988a5395f927894ac5c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T06:49:00.334 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:49:00 smithi038 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.0.service: Main process exited, code=exited, status=137/n/a 2024-03-26T06:49:00.747 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:49:00 smithi038 podman[91583]: 2024-03-26 06:49:00.283856077 +0000 UTC m=+0.017573208 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:49:00.747 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:49:00 smithi038 podman[91583]: 2024-03-26 06:49:00.383538757 +0000 UTC m=+0.117255930 container create db29be44e445651bd2c68d88f34b619be602e5c084742a9f6c46e637f34ad46e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T06:49:00.748 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:49:00 smithi038 podman[91583]: 2024-03-26 06:49:00.619568148 +0000 UTC m=+0.353285266 container init db29be44e445651bd2c68d88f34b619be602e5c084742a9f6c46e637f34ad46e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T06:49:00.748 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:49:00 smithi038 podman[91583]: 2024-03-26 06:49:00.622855 +0000 UTC m=+0.356572125 container start db29be44e445651bd2c68d88f34b619be602e5c084742a9f6c46e637f34ad46e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T06:49:01.084 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:49:00 smithi038 podman[91583]: 2024-03-26 06:49:00.744546664 +0000 UTC m=+0.478263791 container attach db29be44e445651bd2c68d88f34b619be602e5c084742a9f6c46e637f34ad46e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T06:49:01.084 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 26 06:49:00 smithi038 podman[91583]: 2024-03-26 06:49:00.927791381 +0000 UTC m=+0.661508499 container died db29be44e445651bd2c68d88f34b619be602e5c084742a9f6c46e637f34ad46e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T06:49:01.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T06:49:01.527 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:49:01.527 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T06:49:01.527 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T06:49:01.527 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.1 2024-03-26T06:49:01.843 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:01 smithi121 systemd[1]: Stopping Ceph osd.1 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:49:02.108 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:01 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1[40958]: 2024-03-26T06:49:01.839+0000 7f5dc2fc5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:49:02.108 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:01 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1[40958]: 2024-03-26T06:49:01.839+0000 7f5dc2fc5700 -1 osd.1 102 *** Got signal Terminated *** 2024-03-26T06:49:02.108 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:01 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1[40958]: 2024-03-26T06:49:01.839+0000 7f5dc2fc5700 -1 osd.1 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:49:32.147 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:31 smithi121 bash[72349]: time="2024-03-26T06:49:31Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1 in 30 seconds, resorting to SIGKILL" 2024-03-26T06:49:32.147 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:32 smithi121 podman[72349]: 2024-03-26 06:49:32.000821316 +0000 UTC m=+30.280674856 container died f841e6ca2e2923f77c34015950235f491acf1d630b64f032ac46e751eff3cc2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T06:49:32.624 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:32 smithi121 podman[72349]: 2024-03-26 06:49:32.362019522 +0000 UTC m=+30.641873061 container cleanup f841e6ca2e2923f77c34015950235f491acf1d630b64f032ac46e751eff3cc2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-26T06:49:32.625 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:32 smithi121 bash[72349]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1 2024-03-26T06:49:32.896 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:32 smithi121 podman[72380]: 2024-03-26 06:49:32.621713628 +0000 UTC m=+0.621665284 container remove f841e6ca2e2923f77c34015950235f491acf1d630b64f032ac46e751eff3cc2d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f) 2024-03-26T06:49:32.896 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:32 smithi121 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.1.service: Main process exited, code=exited, status=137/n/a 2024-03-26T06:49:33.254 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:32 smithi121 podman[72468]: 2024-03-26 06:49:32.877011606 +0000 UTC m=+0.018233147 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:49:33.254 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:33 smithi121 podman[72468]: 2024-03-26 06:49:33.014292617 +0000 UTC m=+0.155514153 container create c402508225d89816c2361a6210c81cae0222d4193f00b134b95be75b9c3d5127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-26T06:49:33.254 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:33 smithi121 podman[72468]: 2024-03-26 06:49:33.251202649 +0000 UTC m=+0.392424184 container init c402508225d89816c2361a6210c81cae0222d4193f00b134b95be75b9c3d5127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=rishabh-2024mar8-2-7e98f5f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-26T06:49:33.557 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:33 smithi121 podman[72468]: 2024-03-26 06:49:33.254478803 +0000 UTC m=+0.395700330 container start c402508225d89816c2361a6210c81cae0222d4193f00b134b95be75b9c3d5127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f) 2024-03-26T06:49:33.557 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:33 smithi121 podman[72468]: 2024-03-26 06:49:33.325346468 +0000 UTC m=+0.466568009 container attach c402508225d89816c2361a6210c81cae0222d4193f00b134b95be75b9c3d5127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T06:49:33.847 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:33 smithi121 podman[72468]: 2024-03-26 06:49:33.55436776 +0000 UTC m=+0.695589291 container died c402508225d89816c2361a6210c81cae0222d4193f00b134b95be75b9c3d5127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T06:49:34.103 INFO:journalctl@ceph.osd.1.smithi121.stdout:Mar 26 06:49:34 smithi121 podman[72495]: 2024-03-26 06:49:34.086627624 +0000 UTC m=+0.523655226 container remove c402508225d89816c2361a6210c81cae0222d4193f00b134b95be75b9c3d5127 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-2-7e98f5f, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T06:49:34.109 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T06:49:34.151 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:49:34.151 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T06:49:34.152 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T06:49:34.152 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.2 2024-03-26T06:49:34.435 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:34 smithi170 systemd[1]: Stopping Ceph osd.2 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:49:34.697 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2[42088]: 2024-03-26T06:49:34.431+0000 7f4a1d339700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:49:34.697 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2[42088]: 2024-03-26T06:49:34.431+0000 7f4a1d339700 -1 osd.2 102 *** Got signal Terminated *** 2024-03-26T06:49:34.697 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2[42088]: 2024-03-26T06:49:34.431+0000 7f4a1d339700 -1 osd.2 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:49:40.235 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:39 smithi170 podman[74329]: 2024-03-26 06:49:39.929511185 +0000 UTC m=+5.586395942 container died 057b6e93c12be33e151cd1b759c530823373cee15099dafeb7ede828bbee9aa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-26T06:49:40.552 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:40 smithi170 podman[74329]: 2024-03-26 06:49:40.322804254 +0000 UTC m=+5.979689011 container cleanup 057b6e93c12be33e151cd1b759c530823373cee15099dafeb7ede828bbee9aa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T06:49:40.552 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:40 smithi170 bash[74329]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2 2024-03-26T06:49:40.883 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:40 smithi170 podman[74359]: 2024-03-26 06:49:40.549152528 +0000 UTC m=+0.622176316 container remove 057b6e93c12be33e151cd1b759c530823373cee15099dafeb7ede828bbee9aa1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-26T06:49:41.147 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:40 smithi170 podman[74449]: 2024-03-26 06:49:40.783673605 +0000 UTC m=+0.017873913 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:49:41.147 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:40 smithi170 podman[74449]: 2024-03-26 06:49:40.883392741 +0000 UTC m=+0.117593071 container create 3fda6a46d3e1c2c8d4617f184d4abcf190001ce5819d29faefa66ffc49eaeab5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-2-7e98f5f, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-03-26T06:49:41.147 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:41 smithi170 podman[74449]: 2024-03-26 06:49:41.144294908 +0000 UTC m=+0.378495201 container init 3fda6a46d3e1c2c8d4617f184d4abcf190001ce5819d29faefa66ffc49eaeab5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T06:49:41.419 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:41 smithi170 podman[74449]: 2024-03-26 06:49:41.147680336 +0000 UTC m=+0.381880627 container start 3fda6a46d3e1c2c8d4617f184d4abcf190001ce5819d29faefa66ffc49eaeab5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-26T06:49:41.419 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:41 smithi170 podman[74449]: 2024-03-26 06:49:41.261176672 +0000 UTC m=+0.495376973 container attach 3fda6a46d3e1c2c8d4617f184d4abcf190001ce5819d29faefa66ffc49eaeab5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2-deactivate, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-26T06:49:41.742 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:41 smithi170 podman[74449]: 2024-03-26 06:49:41.436184236 +0000 UTC m=+0.670384519 container died 3fda6a46d3e1c2c8d4617f184d4abcf190001ce5819d29faefa66ffc49eaeab5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T06:49:41.998 INFO:journalctl@ceph.osd.2.smithi170.stdout:Mar 26 06:49:41 smithi170 podman[74474]: 2024-03-26 06:49:41.972973172 +0000 UTC m=+0.531704742 container remove 3fda6a46d3e1c2c8d4617f184d4abcf190001ce5819d29faefa66ffc49eaeab5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-26T06:49:42.003 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T06:49:42.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:49:42.050 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T06:49:42.050 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T06:49:42.050 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.3 2024-03-26T06:49:42.334 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:42 smithi038 systemd[1]: Stopping Ceph osd.3 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:49:42.834 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:42 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3[51003]: 2024-03-26T06:49:42.348+0000 7f98d69ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:49:42.835 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:42 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3[51003]: 2024-03-26T06:49:42.348+0000 7f98d69ba700 -1 osd.3 102 *** Got signal Terminated *** 2024-03-26T06:49:42.835 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:42 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3[51003]: 2024-03-26T06:49:42.348+0000 7f98d69ba700 -1 osd.3 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:49:48.245 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:47 smithi038 podman[91701]: 2024-03-26 06:49:47.989421669 +0000 UTC m=+5.744687597 container died 6a7c93c75e7360afb1671dfe8b111d73cf5c2e1ea7f2685f6a2407d64042070f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-26T06:49:48.584 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:48 smithi038 podman[91701]: 2024-03-26 06:49:48.333393054 +0000 UTC m=+6.088658984 container cleanup 6a7c93c75e7360afb1671dfe8b111d73cf5c2e1ea7f2685f6a2407d64042070f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f) 2024-03-26T06:49:48.584 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:48 smithi038 bash[91701]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3 2024-03-26T06:49:48.902 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:48 smithi038 podman[91731]: 2024-03-26 06:49:48.60968887 +0000 UTC m=+0.620880405 container remove 6a7c93c75e7360afb1671dfe8b111d73cf5c2e1ea7f2685f6a2407d64042070f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3, io.buildah.version=1.33.5, ceph=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-26T06:49:49.233 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:48 smithi038 podman[91822]: 2024-03-26 06:49:48.803075291 +0000 UTC m=+0.018445791 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:49:49.233 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:49 smithi038 podman[91822]: 2024-03-26 06:49:49.01911458 +0000 UTC m=+0.234485110 container create 1b503728530fa0506b01089d15a216c6af21e380763cbd19929f52e0682d0743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T06:49:49.539 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:49 smithi038 podman[91822]: 2024-03-26 06:49:49.229990204 +0000 UTC m=+0.445360692 container init 1b503728530fa0506b01089d15a216c6af21e380763cbd19929f52e0682d0743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T06:49:49.539 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:49 smithi038 podman[91822]: 2024-03-26 06:49:49.233206985 +0000 UTC m=+0.448577484 container start 1b503728530fa0506b01089d15a216c6af21e380763cbd19929f52e0682d0743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T06:49:49.540 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:49 smithi038 podman[91822]: 2024-03-26 06:49:49.321738838 +0000 UTC m=+0.537109327 container attach 1b503728530fa0506b01089d15a216c6af21e380763cbd19929f52e0682d0743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f) 2024-03-26T06:49:49.834 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 26 06:49:49 smithi038 podman[91822]: 2024-03-26 06:49:49.536953747 +0000 UTC m=+0.752324240 container died 1b503728530fa0506b01089d15a216c6af21e380763cbd19929f52e0682d0743 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T06:49:50.165 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T06:49:50.208 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:49:50.209 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T06:49:50.209 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T06:49:50.209 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.4 2024-03-26T06:49:50.491 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:49:50 smithi121 systemd[1]: Stopping Ceph osd.4 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:49:50.896 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:49:50 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4[47510]: 2024-03-26T06:49:50.487+0000 7f310ef99700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:49:50.896 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:49:50 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4[47510]: 2024-03-26T06:49:50.487+0000 7f310ef99700 -1 osd.4 102 *** Got signal Terminated *** 2024-03-26T06:49:50.896 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:49:50 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4[47510]: 2024-03-26T06:49:50.487+0000 7f310ef99700 -1 osd.4 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:50:00.334 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:49:59 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:49:59.834+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:00.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:00 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:00.160+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:01.334 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:00 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:00.848+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:01.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:01 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:01.198+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:02.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:01 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:01.792+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:02.334 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:01 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:01.895+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:02.547 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:02 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:02.194+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:03.168 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:02.799+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:03.334 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:02 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:02.862+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:03.561 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:03 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:03.178+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:04.070 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:03 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:03.789+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:04.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:03 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:03.831+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:04.084 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:03 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:03.593+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:04.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:04 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:04.223+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:04.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:04.575+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:04.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:04 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:04.645+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:04.920 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:04 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:04.827+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:05.336 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:04.849+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:05.642 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:05 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:05.638+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:05.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:05 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:05.175+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:05.918 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:05 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:05.846+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:06.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:05 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:05.851+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:06.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:05 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:05.616+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:06.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:06 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:06.193+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:06.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:06 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:06.574+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:07.168 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:06 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:06.682+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:07.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:06 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:06.806+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:07.334 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:06 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:06.852+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:07.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:07 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:07.179+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:08.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:07 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:07.811+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:08.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:07 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:07.593+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:08.168 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:07 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:07.801+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:08.169 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:07 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:07.692+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:08.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:08 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:08.195+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:08.918 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:08 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:08.811+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:08.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:08 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:08.656+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:09.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:08 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:08.843+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:09.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:08 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:08.599+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:09.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:09 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:09.223+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:10.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:09 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:09.796+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:10.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:09 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:09.643+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:10.168 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:09 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:09.688+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:10.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:09 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:09.818+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:10.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:10 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:10.211+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:11.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:10 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:10.835+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:11.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:10 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:10.638+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:11.169 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:10 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:10.704+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:11.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:10 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:10.839+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:11.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:11 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:11.162+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:12.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:11 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:11.836+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:12.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:11 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:11.602+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:12.168 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:11 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:11.670+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:12.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:11 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:11.818+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:12.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:12 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:12.156+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:12.985 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:12 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:12.814+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:12.986 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:12 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:12.615+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:13.168 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:12 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:12.788+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:13.169 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:12 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:12.703+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:13.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:13 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:13.118+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:14.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:13 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:13.794+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:14.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:13 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:13.581+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:14.168 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:13 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:13.813+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:14.169 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:13 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:13.673+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:14.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:14 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:14.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:15.085 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:14 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:14.784+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:15.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:14 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:14.610+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:15.169 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:14 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:14.683+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:15.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:14 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:14.792+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:15.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:15 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:15.109+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:15.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:15 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:15.780+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:15.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:15 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:15.641+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:16.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:15 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:15.787+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:16.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:15 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:15.609+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:16.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:16.144+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:17.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:16 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:16.809+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:17.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:16 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:16.591+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:17.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:16.787+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:17.169 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:16.675+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:17.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:17 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:17.136+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:18.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:17 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:17.794+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:18.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:17 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:17.589+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:18.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:17 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:17.804+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:18.169 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:17 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:17.679+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:18.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:18 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:18.123+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:19.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:18 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:18.800+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:19.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:18 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:18.590+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:19.168 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:18 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:18.783+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:19.169 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:18 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:18.671+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:19.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:19 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:19.140+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:19.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:19 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:19.813+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:19.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:19 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:19.632+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:20.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:19.755+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:20.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:19.604+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:20.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:20 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:20.117+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:20.896 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:20 smithi121 bash[72586]: time="2024-03-26T06:50:20Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4 in 30 seconds, resorting to SIGKILL" 2024-03-26T06:50:20.896 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:20 smithi121 podman[72586]: 2024-03-26 06:50:20.670738242 +0000 UTC m=+30.284263325 container died 3b8039a2d2848a20ee490032d83cadb2331223d50c7599c48f237194cda4779f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T06:50:20.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:20 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:20.655+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:20.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:20 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:20.768+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:21.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:20 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:20.709+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:21.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:20 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:20.592+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:21.296 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:21 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:21.130+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:21.297 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:21 smithi121 podman[72586]: 2024-03-26 06:50:21.062744251 +0000 UTC m=+30.676269330 container cleanup 3b8039a2d2848a20ee490032d83cadb2331223d50c7599c48f237194cda4779f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T06:50:21.297 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:21 smithi121 bash[72586]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4 2024-03-26T06:50:21.625 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:21 smithi121 podman[72620]: 2024-03-26 06:50:21.293994307 +0000 UTC m=+0.620477805 container remove 3b8039a2d2848a20ee490032d83cadb2331223d50c7599c48f237194cda4779f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T06:50:21.625 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:21 smithi121 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.4.service: Main process exited, code=exited, status=137/n/a 2024-03-26T06:50:21.896 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:21 smithi121 podman[72709]: 2024-03-26 06:50:21.525566949 +0000 UTC m=+0.018070240 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:50:21.896 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:21 smithi121 podman[72709]: 2024-03-26 06:50:21.636363314 +0000 UTC m=+0.128866606 container create f9b8d5d450cf6d66281b7ad43006fe87fd6d0922a4c0ebdc72529535970b9aee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T06:50:21.896 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:21 smithi121 podman[72709]: 2024-03-26 06:50:21.855459048 +0000 UTC m=+0.347962336 container init f9b8d5d450cf6d66281b7ad43006fe87fd6d0922a4c0ebdc72529535970b9aee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T06:50:21.897 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:21 smithi121 podman[72709]: 2024-03-26 06:50:21.858621147 +0000 UTC m=+0.351124435 container start f9b8d5d450cf6d66281b7ad43006fe87fd6d0922a4c0ebdc72529535970b9aee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T06:50:21.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:21 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:21.660+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:21.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:21 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:21.743+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:22.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:21 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:21.711+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:22.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:21 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:21.586+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:22.160 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:22 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:22.094+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:22.161 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:21 smithi121 podman[72709]: 2024-03-26 06:50:21.955772157 +0000 UTC m=+0.448275449 container attach f9b8d5d450cf6d66281b7ad43006fe87fd6d0922a4c0ebdc72529535970b9aee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T06:50:22.641 INFO:journalctl@ceph.osd.4.smithi121.stdout:Mar 26 06:50:22 smithi121 podman[72709]: 2024-03-26 06:50:22.158089822 +0000 UTC m=+0.650593110 container died f9b8d5d450cf6d66281b7ad43006fe87fd6d0922a4c0ebdc72529535970b9aee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5) 2024-03-26T06:50:22.729 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T06:50:22.770 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:50:22.770 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T06:50:22.771 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T06:50:22.771 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.5 2024-03-26T06:50:22.834 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:22 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:22.709+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:22.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:22 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:22.562+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:22.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:22 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:22.716+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:22.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:22 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:22.629+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:22.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:22 smithi170 systemd[1]: Stopping Ceph osd.5 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:50:23.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:23 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:23.127+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:23.418 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:23 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:23.064+0000 7f60696d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:50:23.419 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:23 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:23.064+0000 7f60696d8700 -1 osd.5 102 *** Got signal Terminated *** 2024-03-26T06:50:23.419 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:23 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:23.064+0000 7f60696d8700 -1 osd.5 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:50:23.834 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:23 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:23.749+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:23.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:23 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:23.527+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:23.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:23 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:23.601+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:23.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:23 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:23.715+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:24.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:24 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:24.118+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:24.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:24 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:24.480+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:24.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:24 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:24.592+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:24.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:24 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:24.694+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:25.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:24 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:24.756+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:25.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:25 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:25.144+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:25.720 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:25 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:25.454+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:25.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:25 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:25.704+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:25.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:25 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:25.616+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:26.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:25 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:25.716+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:26.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:26.159+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:26.756 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:26 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:26.466+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:26.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:26 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:26.589+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:26.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:26 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:26.753+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:27.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:26 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:26.752+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:27.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:27.120+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:27.834 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:27 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:27.709+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:27.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:27 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:27.489+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:27.919 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5[48956]: 2024-03-26T06:50:27.583+0000 7f60664cd700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:34.047395+0000 front 2024-03-26T06:48:34.047423+0000 (oldest deadline 2024-03-26T06:50:04.252944+0000) 2024-03-26T06:50:27.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:27.784+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:28.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:28 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:28.094+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:28.754 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:28 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:28.475+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:29.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:28 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:28.751+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:29.093 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:28 smithi170 podman[74567]: 2024-03-26 06:50:28.82766213 +0000 UTC m=+5.866599312 container died a3be616570bc17c8cddc2947762f381d6a343a22bf71d015db310b9bf0659ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T06:50:29.093 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:28 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:28.771+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:29.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:29 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:29.141+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:29.418 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:29 smithi170 podman[74567]: 2024-03-26 06:50:29.181078903 +0000 UTC m=+6.220016083 container cleanup a3be616570bc17c8cddc2947762f381d6a343a22bf71d015db310b9bf0659ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-26T06:50:29.419 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:29 smithi170 bash[74567]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5 2024-03-26T06:50:29.727 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:29.431+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:29.784 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:29 smithi170 podman[74584]: 2024-03-26 06:50:29.457446667 +0000 UTC m=+0.624745854 container remove a3be616570bc17c8cddc2947762f381d6a343a22bf71d015db310b9bf0659ab8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T06:50:30.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:29 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:29.723+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:30.096 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:29.781+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:30.097 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:29 smithi170 podman[74680]: 2024-03-26 06:50:29.703270168 +0000 UTC m=+0.018249200 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:50:30.097 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:29 smithi170 podman[74680]: 2024-03-26 06:50:29.808076706 +0000 UTC m=+0.123055734 container create c2a3541147f17712138f4e8d7f6ada53fb8dd811e9ccce0dc0cf94864cb9a53d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T06:50:30.097 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:30 smithi170 podman[74680]: 2024-03-26 06:50:30.010393924 +0000 UTC m=+0.325372963 container init c2a3541147f17712138f4e8d7f6ada53fb8dd811e9ccce0dc0cf94864cb9a53d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5-deactivate, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T06:50:30.097 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:30 smithi170 podman[74680]: 2024-03-26 06:50:30.013486135 +0000 UTC m=+0.328465169 container start c2a3541147f17712138f4e8d7f6ada53fb8dd811e9ccce0dc0cf94864cb9a53d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T06:50:30.097 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:30 smithi170 podman[74680]: 2024-03-26 06:50:30.093887511 +0000 UTC m=+0.408866553 container attach c2a3541147f17712138f4e8d7f6ada53fb8dd811e9ccce0dc0cf94864cb9a53d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:50:30.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:30 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:30.137+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:30.419 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:30 smithi170 podman[74680]: 2024-03-26 06:50:30.30480329 +0000 UTC m=+0.619782353 container died c2a3541147f17712138f4e8d7f6ada53fb8dd811e9ccce0dc0cf94864cb9a53d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-26T06:50:30.698 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:30 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:30.412+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:30.909 INFO:journalctl@ceph.osd.5.smithi170.stdout:Mar 26 06:50:30 smithi170 podman[74706]: 2024-03-26 06:50:30.888751963 +0000 UTC m=+0.578636651 container remove c2a3541147f17712138f4e8d7f6ada53fb8dd811e9ccce0dc0cf94864cb9a53d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T06:50:30.909 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:30 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:30.732+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:30.913 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T06:50:30.955 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:50:30.955 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T06:50:30.956 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T06:50:30.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.6 2024-03-26T06:50:30.987 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:30 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:30.695+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:31.272 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:31 smithi038 systemd[1]: Stopping Ceph osd.6 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:50:31.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:31 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:31.134+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:31.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:31 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:31.401+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:31.584 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:31 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:31.268+0000 7fdaffc3f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:50:31.585 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:31 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:31.268+0000 7fdaffc3f700 -1 osd.6 102 *** Got signal Terminated *** 2024-03-26T06:50:31.585 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:31 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:31.268+0000 7fdaffc3f700 -1 osd.6 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:50:32.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:31 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:31.655+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:32.168 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:31 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:31.762+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:32.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:32 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:32.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:32.650 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:32.388+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:33.084 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:32 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:32.646+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:33.168 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:32 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:32.789+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:33.513 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:33 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:33.185+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:33.513 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:33 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:33.315+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:33.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:33 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:33.420+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:33.835 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:33 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:33.602+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:33.835 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:33 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:33.602+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.555255+0000 front 2024-03-26T06:49:04.555277+0000 (oldest deadline 2024-03-26T06:50:33.357150+0000) 2024-03-26T06:50:33.919 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:33 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:33.641+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:33.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:33 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:33.833+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:34.646 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:34 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:34.345+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:34.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:34 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:34.195+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:34.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:34 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:34.454+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:34.835 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:34 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:34.554+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:34.835 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:34 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:34.554+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.555255+0000 front 2024-03-26T06:49:04.555277+0000 (oldest deadline 2024-03-26T06:50:33.357150+0000) 2024-03-26T06:50:34.919 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:34.642+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:34.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:34.787+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:34.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:34.787+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:35.646 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:35 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:35.309+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:35.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:35 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:35.176+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:35.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:35 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:35.176+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:35.834 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:35 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:35.556+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.450851+0000 front 2024-03-26T06:48:31.450875+0000 (oldest deadline 2024-03-26T06:49:59.453773+0000) 2024-03-26T06:50:35.835 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:35 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6[57874]: 2024-03-26T06:50:35.556+0000 7fdaf8a2c700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.555255+0000 front 2024-03-26T06:49:04.555277+0000 (oldest deadline 2024-03-26T06:50:33.357150+0000) 2024-03-26T06:50:35.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:35 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:35.416+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:35.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:35 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:35.416+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:36.168 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:35 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:35.685+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:36.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:35 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:35.768+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:36.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:35 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:35.768+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:36.646 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:36 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:36.295+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:36.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:36 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:36.149+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:36.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:36 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:36.149+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:36.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:36 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:36.449+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:36.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:36 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:36.449+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:37.139 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:36 smithi038 podman[91942]: 2024-03-26 06:50:36.879797194 +0000 UTC m=+5.719478064 container died 5f6b68aaa1b8c58ce41a33e18af26b551c6ba2f9c33c15d2c7210c6ab83a5730 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True) 2024-03-26T06:50:37.168 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:36 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:36.713+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:37.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:36 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:36.722+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:37.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:36 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:36.722+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:37.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:37 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:37.264+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:37.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:37 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:37.103+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:37.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:37 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:37.103+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:37.433 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:37 smithi038 podman[91942]: 2024-03-26 06:50:37.227546522 +0000 UTC m=+6.067227387 container cleanup 5f6b68aaa1b8c58ce41a33e18af26b551c6ba2f9c33c15d2c7210c6ab83a5730 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f) 2024-03-26T06:50:37.434 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:37 smithi038 bash[91942]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6 2024-03-26T06:50:37.834 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:37 smithi038 podman[91959]: 2024-03-26 06:50:37.637409664 +0000 UTC m=+0.754074568 container remove 5f6b68aaa1b8c58ce41a33e18af26b551c6ba2f9c33c15d2c7210c6ab83a5730 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T06:50:37.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:37 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:37.429+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:37.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:37 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:37.429+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:38.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:37.752+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:38.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:37.753+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:38.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:37.753+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:38.302 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:37 smithi038 podman[92049]: 2024-03-26 06:50:37.996552525 +0000 UTC m=+0.113195512 container create 03daf529490b46267fa88b87ffd60a28497abff966f25c75fb2b9a554e3cf675 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5) 2024-03-26T06:50:38.302 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:37 smithi038 podman[92049]: 2024-03-26 06:50:37.901187835 +0000 UTC m=+0.017830839 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:50:38.302 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:38 smithi038 podman[92049]: 2024-03-26 06:50:38.224234286 +0000 UTC m=+0.340877273 container init 03daf529490b46267fa88b87ffd60a28497abff966f25c75fb2b9a554e3cf675 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T06:50:38.302 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:38 smithi038 podman[92049]: 2024-03-26 06:50:38.22741439 +0000 UTC m=+0.344057377 container start 03daf529490b46267fa88b87ffd60a28497abff966f25c75fb2b9a554e3cf675 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T06:50:38.303 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:38 smithi038 podman[92049]: 2024-03-26 06:50:38.299050634 +0000 UTC m=+0.415693640 container attach 03daf529490b46267fa88b87ffd60a28497abff966f25c75fb2b9a554e3cf675 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T06:50:38.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:38 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:38.246+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:38.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:38 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:38.136+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:38.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:38 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:38.136+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:38.584 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:38 smithi038 podman[92049]: 2024-03-26 06:50:38.528098383 +0000 UTC m=+0.644741373 container died 03daf529490b46267fa88b87ffd60a28497abff966f25c75fb2b9a554e3cf675 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319) 2024-03-26T06:50:38.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:38 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:38.469+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:38.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:38 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:38.469+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:39.013 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:38 smithi038 podman[92076]: 2024-03-26 06:50:38.986338333 +0000 UTC m=+0.453477740 container remove 03daf529490b46267fa88b87ffd60a28497abff966f25c75fb2b9a554e3cf675 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T06:50:39.014 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:39 smithi038 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-26T06:50:39.014 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:39 smithi038 systemd[1]: Stopped Ceph osd.6 for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:50:39.014 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 26 06:50:39 smithi038 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.6.service: Consumed 5min 47.069s CPU time. 2024-03-26T06:50:39.015 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T06:50:39.097 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:50:39.097 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T06:50:39.097 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T06:50:39.098 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.7 2024-03-26T06:50:39.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:38 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:38.722+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:39.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:38 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:38.760+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:39.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:38 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:38.760+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:39.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:39 smithi121 systemd[1]: Stopping Ceph osd.7 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:50:39.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:39.239+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:39.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:39.092+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:39.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:39.092+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:39.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:39 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:39.491+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:39.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:39 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:39.491+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:39.896 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:39.401+0000 7f01e56c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:50:39.896 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:39.401+0000 7f01e56c9700 -1 osd.7 102 *** Got signal Terminated *** 2024-03-26T06:50:39.896 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:39 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:39.401+0000 7f01e56c9700 -1 osd.7 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:50:40.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:39 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:39.793+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:40.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:39 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:39.793+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:40.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:39 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:39.752+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:40.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:40 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:40.286+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:40.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:40 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:40.123+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:40.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:40 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:40.123+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:40.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:40 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:40.468+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:40.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:40 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:40.468+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:41.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:40 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:40.754+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:41.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:40 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:40.754+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:41.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:40 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:40.717+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:41.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:40 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:40.717+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:41.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:41 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:41.278+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:41.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:41 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:41.278+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.864077+0000 front 2024-03-26T06:49:38.863988+0000 (oldest deadline 2024-03-26T06:50:41.164031+0000) 2024-03-26T06:50:41.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:41 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:41.095+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:41.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:41 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:41.095+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:41.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:41 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:41.455+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:41.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:41 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:41.455+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:42.168 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:41 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:41.785+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:42.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:41 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:41.785+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:42.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:41 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:41.713+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:42.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:41 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:41.713+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:42.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:42 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:42.066+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:42.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:42 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:42.066+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:42.397 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:42 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:42.229+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:42.397 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:42 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:42.229+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.864077+0000 front 2024-03-26T06:49:38.863988+0000 (oldest deadline 2024-03-26T06:50:41.164031+0000) 2024-03-26T06:50:42.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:42 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:42.500+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:42.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:42 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:42.500+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:43.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:42 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:42.759+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:43.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:42 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:42.759+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:43.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:42 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:42.750+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:43.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:42 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:42.750+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:43.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:42 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:42.750+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:43.253 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:43 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:43.111+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:43.253 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:43 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:43.111+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:43.253 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:43 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:43.111+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:43.573 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:43 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:43.248+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:43.574 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:43 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:43.248+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.864077+0000 front 2024-03-26T06:49:38.863988+0000 (oldest deadline 2024-03-26T06:50:41.164031+0000) 2024-03-26T06:50:43.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:43 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:43.538+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:43.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:43 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:43.538+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:44.168 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:43 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:43.795+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:44.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:43 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:43.795+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:44.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:43 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:43.778+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:44.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:43 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:43.778+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:44.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:43 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:43.778+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:44.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:44 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:44.228+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.456958+0000 front 2024-03-26T06:49:05.456980+0000 (oldest deadline 2024-03-26T06:50:33.162008+0000) 2024-03-26T06:50:44.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:50:44 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7[54536]: 2024-03-26T06:50:44.228+0000 7f01de4b6700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.864077+0000 front 2024-03-26T06:49:38.863988+0000 (oldest deadline 2024-03-26T06:50:41.164031+0000) 2024-03-26T06:50:44.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:44 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:44.108+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:44.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:44 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:44.108+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:44.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:44 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:44.108+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:44.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:44 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:44.551+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:44.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:44 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:44.551+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:45.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:44 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:44.762+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:45.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:44 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:44.762+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:45.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:44 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:44.762+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:45.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:44 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:44.759+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:45.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:44 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:44.759+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:45.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:45 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:45.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:45.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:45 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:45.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:45.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:45 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:45.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:45.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:45 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:45.554+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:45.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:45 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:45.554+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:45.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:45 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:45.554+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:46.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:45 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:45.794+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:46.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:45 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:45.794+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:46.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:45 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:45.792+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:46.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:45 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:45.792+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:46.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:45 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:45.792+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:46.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:46 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:46.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:46.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:46 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:46.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:46.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:46 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:46.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:46.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:46 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:46.563+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:46.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:46 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:46.563+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:46.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:46 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:46.563+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:47.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:46 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:46.812+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:47.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:46 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:46.812+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:47.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:46 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:46.782+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:47.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:46 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:46.782+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:47.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:46 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:46.782+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:47.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:47 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:47.157+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:47.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:47 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:47.157+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:47.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:47 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:47.157+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:47.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:47 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:47.515+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:47.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:47 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:47.515+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:47.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:47 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:47.515+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:48.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:47 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:47.843+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:48.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:47 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:47.843+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:48.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:47 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:47.805+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:48.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:47 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:47.805+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:48.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:47 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:47.805+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:48.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:48 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:48.173+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:48.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:48 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:48.173+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:48.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:48 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:48.173+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:48.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:48 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:48.474+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:48.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:48 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:48.474+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:48.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:48 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:48.474+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:49.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:48 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:48.833+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:49.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:48 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:48.833+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:49.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:48 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:48.833+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:49.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:48 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:48.822+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:49.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:48 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:48.822+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:49.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:48 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:48.822+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:50:49.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:49 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:49.153+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:49.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:49 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:49.153+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:49.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:49 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:49.153+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:49.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:49 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:49.153+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:50:49.727 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:49 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:49.444+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:49.727 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:49 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:49.444+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:49.727 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:49 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:49.444+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:49.727 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:49 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:49.444+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:50:50.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:49 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:49.820+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:50.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:49 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:49.820+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:50.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:49 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:49.820+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:50.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:49 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:49.775+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:50.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:49 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:49.775+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:50.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:49 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:49.775+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:50:50.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:50 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:50.410+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:50.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:50 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:50.410+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:50.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:50 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:50.410+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:50.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:50 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:50.410+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:50:50.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:50 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:50.196+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:50.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:50 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:50.196+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:50.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:50 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:50.196+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:50.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:50 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:50.196+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:50:51.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:50 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:50.833+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:51.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:50 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:50.833+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:51.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:50 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:50.833+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:51.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:50 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:50.795+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:51.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:50 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:50.795+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:51.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:50 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:50.795+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:50:51.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:51 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:51.178+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:51.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:51 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:51.178+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:51.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:51 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:51.178+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:51.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:51 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:51.178+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:50:51.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:51 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:51.419+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:51.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:51 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:51.419+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:51.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:51 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:51.419+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:51.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:51 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:51.419+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:50:52.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:51 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:51.787+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:52.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:51 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:51.787+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:52.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:51 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:51.787+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:52.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:51 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:51.818+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:52.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:51 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:51.818+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:52.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:51 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:51.818+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:50:52.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:52 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:52.132+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:52.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:52 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:52.132+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:52.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:52 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:52.132+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:52.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:52 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:52.132+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:50:52.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:52 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:52.416+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:52.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:52 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:52.416+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:52.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:52 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:52.416+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:52.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:52 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:52.416+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:50:53.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:52 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:52.829+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:53.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:52 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:52.829+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:53.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:52 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:52.829+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:53.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:52 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:52.809+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:53.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:52 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:52.809+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:53.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:52 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:52.809+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:50:53.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:53 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:53.140+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:53.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:53 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:53.140+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:53.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:53 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:53.140+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:53.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:53 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:53.140+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:50:53.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:53 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:53.379+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:53.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:53 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:53.379+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:53.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:53 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:53.379+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:53.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:53 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:53.379+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:50:54.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:53 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:53.870+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:54.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:53 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:53.870+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:54.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:53 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:53.870+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:54.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:53 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:53.856+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:54.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:53 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:53.857+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:54.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:53 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:53.857+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:50:54.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:54 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:54.106+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:54.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:54 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:54.106+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:54.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:54 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:54.106+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:54.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:54 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:54.106+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:50:54.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:54 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:54.344+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:54.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:54 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:54.344+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:54.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:54 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:54.344+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:54.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:54 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:54.344+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:50:55.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:54 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:54.903+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:55.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:54 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:54.903+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:55.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:54 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:54.903+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:55.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:54 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:54.882+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:55.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:54 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:54.882+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:55.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:54 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:54.882+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:50:55.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:55 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:55.083+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:55.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:55 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:55.083+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:55.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:55 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:55.083+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:55.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:55 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:55.083+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:50:55.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:55 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:55.332+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:55.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:55 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:55.332+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:55.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:55 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:55.332+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:55.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:55 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:55.332+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:50:56.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:55 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:55.944+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:56.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:55 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:55.944+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:56.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:55 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:55.944+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:56.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:55 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:55.879+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:56.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:55 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:55.879+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:56.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:55 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:55.879+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:50:56.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:56 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:56.076+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:56.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:56 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:56.076+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:56.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:56 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:56.076+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:56.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:56 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:56.076+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:50:56.834 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:56 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:56.357+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:56.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:56 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:56.357+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:56.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:56 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:56.357+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:56.835 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:56 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:56.357+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:50:57.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:56 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:56.916+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:57.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:56 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:56.916+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:57.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:56 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:56.916+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:57.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:56 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:56.868+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:57.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:56 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:56.868+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:57.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:56 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:56.868+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:50:57.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:57 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:57.060+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:57.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:57 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:57.060+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:57.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:57 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:57.060+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:57.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:57 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:57.060+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:50:57.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:57 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:57.319+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:57.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:57 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:57.319+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:57.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:57 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:57.319+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:57.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:57 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:57.319+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:50:58.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:57 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:57.867+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:58.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:57 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:57.867+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:58.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:57 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:57.867+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:58.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:57 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:57.906+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:58.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:57 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:57.906+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:58.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:57 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:57.906+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:50:58.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:58 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:58.082+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:58.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:58 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:58.082+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:58.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:58 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:58.082+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:58.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:58 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:58.082+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:50:58.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:58 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:58.312+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:58.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:58 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:58.312+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:58.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:58 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:58.312+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:58.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:58 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:58.312+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:50:59.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:58 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:58.908+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:50:59.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:58 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:58.908+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:50:59.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:58 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:58.908+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:50:59.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:58 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:58.896+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:50:59.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:58 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:58.896+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:50:59.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:58 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:58.896+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:50:59.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:59 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:59.089+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:50:59.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:59 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:59.089+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:50:59.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:59 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:59.089+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:50:59.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:50:59 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:50:59.089+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:50:59.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:59 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:59.323+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:50:59.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:59 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:59.323+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:50:59.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:59 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:59.323+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:50:59.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:50:59 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:50:59.323+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:00.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:59 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:59.913+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:00.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:59 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:59.913+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:00.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:50:59 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:50:59.913+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:00.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:59 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:59.905+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:00.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:59 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:59.905+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:00.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:50:59 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:50:59.905+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:00.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:00 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:00.135+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:00.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:00 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:00.135+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:00.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:00 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:00.135+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:00.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:00 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:00.135+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:00.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:00 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:00.289+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:00.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:00 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:00.289+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:00.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:00 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:00.289+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:00.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:00 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:00.289+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:01.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:01 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:01.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:01.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:01 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:01.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:01.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:01 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:01.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:01.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:01 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:01.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:01.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:00 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:00.954+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:01.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:00 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:00.954+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:01.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:00 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:00.954+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:01.419 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:00 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:00.923+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:01.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:00 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:00.923+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:01.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:00 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:00.923+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:01.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:01 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:01.240+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:01.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:01 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:01.240+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:01.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:01 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:01.240+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:01.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:01 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:01.240+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:02.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:02 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:02.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:02.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:02 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:02.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:02.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:02 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:02.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:02.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:02 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:02.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:02.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:01 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:01.994+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:02.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:01 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:01.994+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:02.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:01 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:01.994+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:02.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:01 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:01.962+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:02.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:01 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:01.962+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:02.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:01 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:01.962+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:02.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:02 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:02.282+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:02.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:02 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:02.282+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:02.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:02 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:02.282+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:02.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:02 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:02.282+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:03.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:02.977+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:03.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:02.977+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:03.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:02.977+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:03.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:02.921+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:03.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:02.921+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:03.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:02 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:02.921+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:03.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:03 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:03.254+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:03.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:03 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:03.254+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:03.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:03 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:03.254+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:03.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:03 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:03.254+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:03.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:03 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:03.182+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:03.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:03 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:03.182+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:03.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:03 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:03.182+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:03.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:03 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:03.182+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:04.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:04 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:04.003+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:04.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:04 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:04.003+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:04.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:04 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:04.003+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:04.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:03 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:03.895+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:04.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:03 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:03.895+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:04.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:03 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:03.895+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:04.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:04.293+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:04.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:04.293+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:04.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:04.293+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:04.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:04 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:04.293+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:04.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:04 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:04.214+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:04.648 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:04 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:04.214+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:04.648 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:04 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:04.214+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:04.648 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:04 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:04.214+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:05.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:05 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:05.033+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:05.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:05 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:05.033+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:05.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:05 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:05.033+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:05.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:04 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:04.883+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:05.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:04 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:04.883+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:05.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:04 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:04.883+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:05.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:05 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:05.258+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:05.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:05 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:05.258+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:05.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:05 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:05.258+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:05.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:05 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:05.258+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:05.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:05 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:05.192+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:05.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:05 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:05.192+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:05.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:05 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:05.192+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:05.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:05 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:05.192+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:06.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:06 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:06.017+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:06.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:06 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:06.017+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:06.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:06 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:06.017+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:06.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:05 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:05.834+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:06.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:05 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:05.834+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:06.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:05 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:05.834+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:06.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:06 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:06.252+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:06.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:06 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:06.252+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:06.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:06 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:06.252+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:06.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:06 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:06.252+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:06.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:06 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:06.156+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:06.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:06 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:06.156+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:06.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:06 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:06.156+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:06.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:06 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:06.156+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:07.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:07 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:07.027+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:07.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:07 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:07.027+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:07.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:07 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:07.027+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:07.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:06 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:06.825+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:07.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:06 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:06.825+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:07.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:06 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:06.825+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:07.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:07 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:07.226+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:07.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:07 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:07.226+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:07.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:07 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:07.226+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:07.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:07 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:07.226+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:07.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:07 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:07.147+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:07.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:07 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:07.147+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:07.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:07 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:07.147+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:07.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:07 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:07.147+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:07.951 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:51:07 smithi121 podman[72826]: 2024-03-26 06:51:07.652678685 +0000 UTC m=+28.373486495 container died cadb09d084a81d3788622dd0e3f8f53664c5708f1dd425c5817e32320de33f66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T06:51:08.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:08 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:08.019+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:08.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:08 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:08.019+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:08.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:08 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:08.019+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:08.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:07 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:07.848+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:08.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:07 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:07.848+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:08.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:07 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:07.848+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:08.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:51:07 smithi121 podman[72826]: 2024-03-26 06:51:07.948593906 +0000 UTC m=+28.669401699 container cleanup cadb09d084a81d3788622dd0e3f8f53664c5708f1dd425c5817e32320de33f66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T06:51:08.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:51:07 smithi121 bash[72826]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7 2024-03-26T06:51:08.396 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:51:08 smithi121 podman[72846]: 2024-03-26 06:51:08.107977929 +0000 UTC m=+0.450546000 container remove cadb09d084a81d3788622dd0e3f8f53664c5708f1dd425c5817e32320de33f66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T06:51:08.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:08 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:08.135+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:08.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:08 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:08.135+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:08.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:08 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:08.135+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:08.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:08 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:08.135+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:08.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:08 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:08.243+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:08.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:08 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:08.243+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:08.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:08 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:08.243+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:08.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:08 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:08.243+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:08.814 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:51:08 smithi121 podman[72935]: 2024-03-26 06:51:08.401025677 +0000 UTC m=+0.017442733 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:51:08.814 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:51:08 smithi121 podman[72935]: 2024-03-26 06:51:08.500617953 +0000 UTC m=+0.117035019 container create 5b52758131725231f3bfda7c2255f15e11ffe3d70cf0033482c13cc0ed22475a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T06:51:08.814 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:51:08 smithi121 podman[72935]: 2024-03-26 06:51:08.711285951 +0000 UTC m=+0.327702999 container init 5b52758131725231f3bfda7c2255f15e11ffe3d70cf0033482c13cc0ed22475a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T06:51:08.814 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:51:08 smithi121 podman[72935]: 2024-03-26 06:51:08.714422117 +0000 UTC m=+0.330839159 container start 5b52758131725231f3bfda7c2255f15e11ffe3d70cf0033482c13cc0ed22475a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-26T06:51:08.814 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:51:08 smithi121 podman[72935]: 2024-03-26 06:51:08.811084847 +0000 UTC m=+0.427501912 container attach 5b52758131725231f3bfda7c2255f15e11ffe3d70cf0033482c13cc0ed22475a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True) 2024-03-26T06:51:09.146 INFO:journalctl@ceph.osd.7.smithi121.stdout:Mar 26 06:51:09 smithi121 podman[72935]: 2024-03-26 06:51:09.001727124 +0000 UTC m=+0.618144169 container died 5b52758131725231f3bfda7c2255f15e11ffe3d70cf0033482c13cc0ed22475a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-26T06:51:09.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:09 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:09.008+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:09.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:09 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:09.008+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:09.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:09 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:09.008+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:09.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:08 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:08.882+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:09.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:08 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:08.882+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:09.170 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:08 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:08.882+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:09.439 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:09 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:09.175+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:09.439 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:09 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:09.175+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:09.439 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:09 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:09.175+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:09.439 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:09 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:09.175+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:09.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:09 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:09.206+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:09.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:09 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:09.206+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:09.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:09 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:09.206+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:09.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:09 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:09.206+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:09.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T06:51:09.637 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:51:09.637 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T06:51:09.638 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-26T06:51:09.638 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.8 2024-03-26T06:51:09.918 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:09 smithi170 systemd[1]: Stopping Ceph osd.8 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:51:10.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:10 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:10.041+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:10.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:10 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:10.041+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:10.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:10 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:10.041+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:10.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:09 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:09.928+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:10.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:09 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:09.928+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:10.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:09 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:09.928+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:10.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:09 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:09.956+0000 7f8c1e2b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:51:10.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:09 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:09.956+0000 7f8c1e2b9700 -1 osd.8 102 *** Got signal Terminated *** 2024-03-26T06:51:10.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:09 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:09.956+0000 7f8c1e2b9700 -1 osd.8 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:51:10.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:10 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:10.190+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:10.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:10 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:10.190+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:10.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:10 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:10.190+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:10.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:10 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:10.190+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:10.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:10 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:10.151+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:10.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:10 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:10.151+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:10.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:10 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:10.151+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:10.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:10 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:10.151+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:11.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:11 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:11.001+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:11.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:11 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:11.001+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:11.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:11 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:11.001+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:11.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:10 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:10.927+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:11.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:10 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:10.927+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:11.420 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:10 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:10.927+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:11.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:11 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:11.211+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:11.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:11 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:11.211+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:11.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:11 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:11.211+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:11.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:11 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:11.211+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:11.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:11 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:11.177+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:11.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:11 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:11.177+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:11.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:11 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:11.177+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:11.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:11 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:11.177+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:12.419 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:11 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:11.924+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:12.419 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:11 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:11.924+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:12.419 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:11 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:11.924+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:12.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:11 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:11.955+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:12.420 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:11 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:11.955+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:12.420 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:11 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:11.955+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:12.486 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:12 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:12.167+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:12.486 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:12 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:12.167+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:12.486 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:12 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:12.167+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:12.486 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:12 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:12.167+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:12.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:12 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:12.196+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:12.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:12 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:12.196+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:12.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:12 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:12.196+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:12.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:12 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:12.196+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:13.419 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:12 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:12.947+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:13.419 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:12 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:12.947+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:13.419 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:12 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:12.947+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:13.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:12 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:12.916+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:13.420 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:12 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:12.916+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:13.420 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:12 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:12.916+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:13.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:13 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:13.133+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:13.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:13 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:13.133+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:13.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:13 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:13.133+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:13.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:13 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:13.133+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:13.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:13 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:13.155+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:13.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:13 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:13.155+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:13.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:13 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:13.155+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:13.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:13 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:13.155+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:14.419 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:13 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:13.924+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:14.419 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:13 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:13.924+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:14.419 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:13 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:13.924+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:14.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:13 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:13.918+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:14.420 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:13 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:13.918+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:14.420 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:13 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:13.918+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:14.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:14 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:14.130+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:14.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:14 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:14.130+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:14.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:14 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:14.130+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:14.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:14 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:14.130+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:14.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:14 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:14.158+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:14.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:14 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:14.158+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:14.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:14 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:14.158+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:14.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:14 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:14.158+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:15.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:14 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:14.886+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.514149+0000 front 2024-03-26T06:49:05.514202+0000 (oldest deadline 2024-03-26T06:50:33.518047+0000) 2024-03-26T06:51:15.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:14 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:14.886+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:35.218972+0000 front 2024-03-26T06:49:35.219192+0000 (oldest deadline 2024-03-26T06:50:40.519117+0000) 2024-03-26T06:51:15.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:14 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8[55562]: 2024-03-26T06:51:14.886+0000 7f8c170a6700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:46.721407+0000 front 2024-03-26T06:49:46.721444+0000 (oldest deadline 2024-03-26T06:50:48.421308+0000) 2024-03-26T06:51:15.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:14 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:14.900+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:15.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:14 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:14.900+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:15.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:14 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:14.900+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:15.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:15 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:15.168+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:15.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:15 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:15.168+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:15.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:15 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:15.168+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:15.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:15 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:15.168+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:15.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:15 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:15.147+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:15.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:15 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:15.147+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:15.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:15 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:15.147+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:15.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:15 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:15.147+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:15.909 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:15 smithi170 podman[74798]: 2024-03-26 06:51:15.603914566 +0000 UTC m=+5.771240314 container died c9464b613a64d1b5b677539a3830e21aeb3352516ba4056b005f16dbc500bc4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T06:51:16.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:15 smithi170 podman[74798]: 2024-03-26 06:51:15.998038441 +0000 UTC m=+6.165364172 container cleanup c9464b613a64d1b5b677539a3830e21aeb3352516ba4056b005f16dbc500bc4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-26T06:51:16.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:15 smithi170 bash[74798]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8 2024-03-26T06:51:16.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:15 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:15.931+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:16.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:15 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:15.931+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:16.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:15 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:15.931+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:16.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:16.104+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:16.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:16.104+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:16.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:16.104+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:16.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:16 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:16.104+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:16.578 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:16 smithi170 podman[74816]: 2024-03-26 06:51:16.265918363 +0000 UTC m=+0.664094545 container remove c9464b613a64d1b5b677539a3830e21aeb3352516ba4056b005f16dbc500bc4a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:51:16.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:16 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:16.126+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:16.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:16 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:16.126+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:16.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:16 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:16.126+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:16.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:16 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:16.126+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:16.832 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:16 smithi170 podman[74903]: 2024-03-26 06:51:16.575227815 +0000 UTC m=+0.087470789 container create c590310eb862195a48de67e54322e2d4e0d52fb56c0cf37b252a4a8e70aa3764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T06:51:16.833 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:16 smithi170 podman[74903]: 2024-03-26 06:51:16.505437043 +0000 UTC m=+0.017680022 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:51:16.833 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:16 smithi170 podman[74903]: 2024-03-26 06:51:16.744572866 +0000 UTC m=+0.256815841 container init c590310eb862195a48de67e54322e2d4e0d52fb56c0cf37b252a4a8e70aa3764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-2-7e98f5f) 2024-03-26T06:51:16.833 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:16 smithi170 podman[74903]: 2024-03-26 06:51:16.747874062 +0000 UTC m=+0.260117030 container start c590310eb862195a48de67e54322e2d4e0d52fb56c0cf37b252a4a8e70aa3764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:51:16.833 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:16 smithi170 podman[74903]: 2024-03-26 06:51:16.828783807 +0000 UTC m=+0.341026786 container attach c590310eb862195a48de67e54322e2d4e0d52fb56c0cf37b252a4a8e70aa3764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T06:51:17.169 INFO:journalctl@ceph.osd.8.smithi170.stdout:Mar 26 06:51:17 smithi170 podman[74903]: 2024-03-26 06:51:17.035678278 +0000 UTC m=+0.547921250 container died c590310eb862195a48de67e54322e2d4e0d52fb56c0cf37b252a4a8e70aa3764 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T06:51:17.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:16.961+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:17.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:16.961+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:17.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:16 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:16.961+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:17.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:17 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:17.120+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:17.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:17 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:17.120+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:17.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:17 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:17.120+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:17.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:17 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:17.120+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:17.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:17 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:17.152+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:17.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:17 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:17.152+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:17.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:17 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:17.152+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:17.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:17 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:17.152+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:17.595 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.8.service' 2024-03-26T06:51:17.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:51:17.636 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-26T06:51:17.636 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-26T06:51:17.637 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.9 2024-03-26T06:51:17.933 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:17 smithi038 systemd[1]: Stopping Ceph osd.9 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:51:17.933 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:17 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:17.928+0000 7fa3db0c8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:51:17.933 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:17 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:17.928+0000 7fa3db0c8700 -1 osd.9 102 *** Got signal Terminated *** 2024-03-26T06:51:18.239 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:17 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:17.915+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:18.240 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:17 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:17.915+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:18.240 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:17 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:17.915+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:18.334 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:17 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:17.928+0000 7fa3db0c8700 -1 osd.9 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:51:18.334 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:18 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:18.106+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:18.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:18 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:18.106+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:18.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:18 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:18.106+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:18.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:18 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:18.106+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:18.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:18 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:18.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:18.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:18 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:18.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:18.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:18 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:18.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:18.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:18 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:18.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:19.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:19 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:19.096+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:19.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:19 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:19.096+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:19.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:19 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:19.096+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:19.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:19 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:19.096+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:19.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:18 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:18.931+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:19.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:18 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:18.931+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:19.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:18 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:18.931+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:19.584 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:19.095+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:19.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:19.095+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:19.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:19.095+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:19.585 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:19 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:19.095+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:20.334 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:20 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:20.064+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:20.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:20 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:20.064+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:20.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:20 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:20.064+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:20.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:20 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:20.064+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:20.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:20 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:20.102+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:20.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:20 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:20.102+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:20.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:20 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:20.102+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:20.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:20 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:20.102+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:20.418 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:19 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:19.964+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:20.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:19 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:19.964+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:20.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:19 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:19.964+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:21.334 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:21 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:21.063+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:21.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:21 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:21.063+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:21.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:21 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:21.063+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:21.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:21 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:21.063+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:21.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:21 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:21.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:21.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:21 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:21.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:21.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:21 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:21.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:21.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:21 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:21.139+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:21.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:20 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:20.931+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:21.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:20 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:20.931+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:21.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:20 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:20.931+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:22.334 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:22 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:22.031+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:33.081790+0000 front 2024-03-26T06:48:33.081887+0000 (oldest deadline 2024-03-26T06:50:03.084211+0000) 2024-03-26T06:51:22.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:22 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:22.032+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:04.185181+0000 front 2024-03-26T06:49:04.185213+0000 (oldest deadline 2024-03-26T06:50:34.687708+0000) 2024-03-26T06:51:22.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:22 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:22.032+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:39.388538+0000 front 2024-03-26T06:49:39.388578+0000 (oldest deadline 2024-03-26T06:50:44.688502+0000) 2024-03-26T06:51:22.335 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:22 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9[64918]: 2024-03-26T06:51:22.032+0000 7fa3d7ebd700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:44.688806+0000 front 2024-03-26T06:49:44.688842+0000 (oldest deadline 2024-03-26T06:50:49.388879+0000) 2024-03-26T06:51:22.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:22 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:22.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:22.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:22 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:22.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:22.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:22 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:22.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:22.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:22 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:22.121+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:22.418 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:21 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:21.922+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:22.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:21 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:21.922+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:22.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:21 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:21.922+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:23.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:22 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:22.913+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:23.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:22 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:22.913+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:23.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:22 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:22.913+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:23.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:22 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:22.913+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:23.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:23 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:23.134+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:23.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:23 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:23.134+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:23.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:23 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:23.134+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:23.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:23 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:23.134+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:23.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:23 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:23.134+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:55.034274+0000 front 2024-03-26T06:49:55.034320+0000 (oldest deadline 2024-03-26T06:51:22.337378+0000) 2024-03-26T06:51:23.667 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:23 smithi038 podman[92167]: 2024-03-26 06:51:23.402391222 +0000 UTC m=+5.569232127 container died 7f96f26d1113a933eee1ddc376d2bfd14f6d216568be166561dec136767be22d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T06:51:24.084 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:23 smithi038 podman[92167]: 2024-03-26 06:51:23.755176795 +0000 UTC m=+5.922017694 container cleanup 7f96f26d1113a933eee1ddc376d2bfd14f6d216568be166561dec136767be22d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-26T06:51:24.085 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:23 smithi038 bash[92167]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9 2024-03-26T06:51:24.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:24 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:24.113+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:24.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:24 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:24.113+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:24.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:24 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:24.113+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:24.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:24 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:24.113+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:24.397 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:24 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:24.113+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:55.034274+0000 front 2024-03-26T06:49:55.034320+0000 (oldest deadline 2024-03-26T06:51:22.337378+0000) 2024-03-26T06:51:24.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:23 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:23.924+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:24.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:23 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:23.924+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:24.420 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:23 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:23.924+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:24.420 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:23 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:23.924+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:24.475 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:24 smithi038 podman[92186]: 2024-03-26 06:51:24.139739458 +0000 UTC m=+0.733335858 container remove 7f96f26d1113a933eee1ddc376d2bfd14f6d216568be166561dec136767be22d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T06:51:24.746 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:24 smithi038 podman[92277]: 2024-03-26 06:51:24.375665103 +0000 UTC m=+0.017594346 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:51:24.746 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:24 smithi038 podman[92277]: 2024-03-26 06:51:24.482201092 +0000 UTC m=+0.124130319 container create c4e3e28a2087eb421a627ee792e593b7cb7f5485df6c5c75981f7deb35edadda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T06:51:24.746 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:24 smithi038 podman[92277]: 2024-03-26 06:51:24.743248753 +0000 UTC m=+0.385177980 container init c4e3e28a2087eb421a627ee792e593b7cb7f5485df6c5c75981f7deb35edadda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-26T06:51:25.047 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:24 smithi038 podman[92277]: 2024-03-26 06:51:24.746462907 +0000 UTC m=+0.388392138 container start c4e3e28a2087eb421a627ee792e593b7cb7f5485df6c5c75981f7deb35edadda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T06:51:25.048 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:24 smithi038 podman[92277]: 2024-03-26 06:51:24.826650346 +0000 UTC m=+0.468579588 container attach c4e3e28a2087eb421a627ee792e593b7cb7f5485df6c5c75981f7deb35edadda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T06:51:25.326 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 26 06:51:25 smithi038 podman[92277]: 2024-03-26 06:51:25.0452162 +0000 UTC m=+0.687145426 container died c4e3e28a2087eb421a627ee792e593b7cb7f5485df6c5c75981f7deb35edadda (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T06:51:25.418 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:24 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:24.938+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:25.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:24 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:24.939+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:25.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:24 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:24.939+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:25.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:24 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:24.939+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:25.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:25 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:25.158+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:25.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:25 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:25.158+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:25.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:25 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:25.158+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:25.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:25 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:25.158+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:25.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:25 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:25.158+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:55.034274+0000 front 2024-03-26T06:49:55.034320+0000 (oldest deadline 2024-03-26T06:51:22.337378+0000) 2024-03-26T06:51:25.828 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.9.service' 2024-03-26T06:51:25.868 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:51:25.868 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-26T06:51:25.868 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-26T06:51:25.869 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.10 2024-03-26T06:51:26.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:25 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:25.898+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:26.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:25 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:25.898+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:26.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:25 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:25.898+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:26.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:25 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:25.898+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:26.170 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:25 smithi121 systemd[1]: Stopping Ceph osd.10 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:51:26.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:26.165+0000 7f08b8be7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:51:26.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:26.165+0000 7f08b8be7700 -1 osd.10 102 *** Got signal Terminated *** 2024-03-26T06:51:26.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:26.165+0000 7f08b8be7700 -1 osd.10 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:51:26.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:26.179+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:26.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:26.179+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:26.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:26.179+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:26.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:26.179+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:26.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:26 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:26.179+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:55.034274+0000 front 2024-03-26T06:49:55.034320+0000 (oldest deadline 2024-03-26T06:51:22.337378+0000) 2024-03-26T06:51:27.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:26 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:26.864+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:27.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:26 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:26.864+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:27.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:26 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:26.864+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:27.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:26 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:26.864+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:27.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:27.224+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:27.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:27.224+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:27.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:27.224+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:27.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:27.224+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:27.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:27 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:27.224+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:55.034274+0000 front 2024-03-26T06:49:55.034320+0000 (oldest deadline 2024-03-26T06:51:22.337378+0000) 2024-03-26T06:51:28.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:27.886+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:28.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:27.886+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:28.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:27.886+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:28.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:27 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:27.886+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:28.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:28 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:28.246+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:28.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:28 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:28.246+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:28.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:28 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:28.246+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:28.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:28 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:28.246+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:28.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:28 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:28.246+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:55.034274+0000 front 2024-03-26T06:49:55.034320+0000 (oldest deadline 2024-03-26T06:51:22.337378+0000) 2024-03-26T06:51:29.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:28 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:28.933+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:29.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:28 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:28.933+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:29.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:28 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:28.933+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:29.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:28 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:28.933+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:29.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:29 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:29.288+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:29.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:29 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:29.288+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:29.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:29 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:29.288+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:29.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:29 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:29.288+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:29.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:29 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:29.288+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:55.034274+0000 front 2024-03-26T06:49:55.034320+0000 (oldest deadline 2024-03-26T06:51:22.337378+0000) 2024-03-26T06:51:30.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:29.895+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:30.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:29.895+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:30.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:29.895+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:30.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:29.895+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:30.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:29.895+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6812 osd.5 since back 2024-03-26T06:50:27.769952+0000 front 2024-03-26T06:50:27.770058+0000 (oldest deadline 2024-03-26T06:51:29.370468+0000) 2024-03-26T06:51:30.646 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:30 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:30.257+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:31.923981+0000 front 2024-03-26T06:48:31.923916+0000 (oldest deadline 2024-03-26T06:49:59.327721+0000) 2024-03-26T06:51:30.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:30 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:30.257+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:05.729284+0000 front 2024-03-26T06:49:05.729369+0000 (oldest deadline 2024-03-26T06:50:34.531834+0000) 2024-03-26T06:51:30.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:30 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:30.257+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:36.232562+0000 front 2024-03-26T06:49:36.232591+0000 (oldest deadline 2024-03-26T06:50:42.132584+0000) 2024-03-26T06:51:30.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:30 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:30.257+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.38:6814 osd.3 since back 2024-03-26T06:49:45.633722+0000 front 2024-03-26T06:49:45.633774+0000 (oldest deadline 2024-03-26T06:50:49.133665+0000) 2024-03-26T06:51:30.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:30 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:30.257+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:55.034274+0000 front 2024-03-26T06:49:55.034320+0000 (oldest deadline 2024-03-26T06:51:22.337378+0000) 2024-03-26T06:51:30.647 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:30 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10[61496]: 2024-03-26T06:51:30.257+0000 7f08b59dc700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.170:6812 osd.5 since back 2024-03-26T06:50:24.038262+0000 front 2024-03-26T06:50:24.038298+0000 (oldest deadline 2024-03-26T06:51:29.338101+0000) 2024-03-26T06:51:31.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:30 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:30.941+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:31.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:30 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:30.941+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:31.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:30 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:30.941+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:31.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:30 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:30.941+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:31.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:30 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:30.941+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6812 osd.5 since back 2024-03-26T06:50:27.769952+0000 front 2024-03-26T06:50:27.770058+0000 (oldest deadline 2024-03-26T06:51:29.370468+0000) 2024-03-26T06:51:32.169 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:31 smithi121 podman[73053]: 2024-03-26 06:51:31.898343886 +0000 UTC m=+5.835804716 container died 88d06d00d8924212770bfb45ffffb1988c6136f5f8ec9ead1965c9ede85ce293 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-26T06:51:32.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:31 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:31.966+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:32.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:31 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:31.966+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:32.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:31 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:31.966+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:32.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:31 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:31.966+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:32.420 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:31 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:31.966+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6812 osd.5 since back 2024-03-26T06:50:27.769952+0000 front 2024-03-26T06:50:27.770058+0000 (oldest deadline 2024-03-26T06:51:29.370468+0000) 2024-03-26T06:51:32.477 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:32 smithi121 podman[73053]: 2024-03-26 06:51:32.26486227 +0000 UTC m=+6.202323082 container cleanup 88d06d00d8924212770bfb45ffffb1988c6136f5f8ec9ead1965c9ede85ce293 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T06:51:32.478 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:32 smithi121 bash[73053]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10 2024-03-26T06:51:32.770 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:32 smithi121 podman[73071]: 2024-03-26 06:51:32.474447623 +0000 UTC m=+0.574601418 container remove 88d06d00d8924212770bfb45ffffb1988c6136f5f8ec9ead1965c9ede85ce293 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319) 2024-03-26T06:51:33.039 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:32 smithi121 podman[73158]: 2024-03-26 06:51:32.767116765 +0000 UTC m=+0.092664631 container create dc71bdf418e9fdc075506a123c1fdf7a7738e5858c8e621f1a465def88efd5fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10-deactivate, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-26T06:51:33.039 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:32 smithi121 podman[73158]: 2024-03-26 06:51:32.691828963 +0000 UTC m=+0.017376836 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:51:33.039 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:32 smithi121 podman[73158]: 2024-03-26 06:51:32.969392355 +0000 UTC m=+0.294940234 container init dc71bdf418e9fdc075506a123c1fdf7a7738e5858c8e621f1a465def88efd5fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-26T06:51:33.040 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:32 smithi121 podman[73158]: 2024-03-26 06:51:32.972435199 +0000 UTC m=+0.297983083 container start dc71bdf418e9fdc075506a123c1fdf7a7738e5858c8e621f1a465def88efd5fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=rishabh-2024mar8-2-7e98f5f) 2024-03-26T06:51:33.040 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:33 smithi121 podman[73158]: 2024-03-26 06:51:33.035849889 +0000 UTC m=+0.361397765 container attach dc71bdf418e9fdc075506a123c1fdf7a7738e5858c8e621f1a465def88efd5fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:51:33.396 INFO:journalctl@ceph.osd.10.smithi121.stdout:Mar 26 06:51:33 smithi121 podman[73158]: 2024-03-26 06:51:33.265869427 +0000 UTC m=+0.591417313 container died dc71bdf418e9fdc075506a123c1fdf7a7738e5858c8e621f1a465def88efd5fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-03-26T06:51:33.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:32 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:32.936+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:33.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:32 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:32.936+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:33.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:32 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:32.936+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:33.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:32 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:32.936+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:33.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:32 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:32.936+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6812 osd.5 since back 2024-03-26T06:50:27.769952+0000 front 2024-03-26T06:50:27.770058+0000 (oldest deadline 2024-03-26T06:51:29.370468+0000) 2024-03-26T06:51:33.802 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.10.service' 2024-03-26T06:51:33.843 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:51:33.843 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-26T06:51:33.843 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-26T06:51:33.843 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.11 2024-03-26T06:51:34.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:33 smithi170 systemd[1]: Stopping Ceph osd.11 for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:51:34.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:33 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:33.950+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:34.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:33 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:33.950+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:34.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:33 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:33.950+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:34.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:33 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:33.950+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:34.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:33 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:33.950+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6812 osd.5 since back 2024-03-26T06:50:27.769952+0000 front 2024-03-26T06:50:27.770058+0000 (oldest deadline 2024-03-26T06:51:29.370468+0000) 2024-03-26T06:51:34.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:34.426+0000 7f1f22dea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:51:34.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:34.426+0000 7f1f22dea700 -1 osd.11 102 *** Got signal Terminated *** 2024-03-26T06:51:34.919 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:34.426+0000 7f1f22dea700 -1 osd.11 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T06:51:35.358 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:34.991+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:35.359 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:34.991+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:35.359 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:34.991+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:35.359 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:34.991+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:35.359 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:34 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:34.991+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6812 osd.5 since back 2024-03-26T06:50:27.769952+0000 front 2024-03-26T06:50:27.770058+0000 (oldest deadline 2024-03-26T06:51:29.370468+0000) 2024-03-26T06:51:36.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:35 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:35.967+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:36.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:35 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:35.967+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:36.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:35 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:35.967+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:36.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:35 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:35.967+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:36.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:35 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:35.967+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6812 osd.5 since back 2024-03-26T06:50:27.769952+0000 front 2024-03-26T06:50:27.770058+0000 (oldest deadline 2024-03-26T06:51:29.370468+0000) 2024-03-26T06:51:37.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:36 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:36.920+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:37.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:36 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:36.920+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:37.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:36 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:36.920+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:37.419 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:36 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:36.920+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:37.420 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:36 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:36.920+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6812 osd.5 since back 2024-03-26T06:50:27.769952+0000 front 2024-03-26T06:50:27.770058+0000 (oldest deadline 2024-03-26T06:51:29.370468+0000) 2024-03-26T06:51:38.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:37.874+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:38.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:37.874+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:38.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:37.874+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:38.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:37.874+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:38.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:37.874+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6812 osd.5 since back 2024-03-26T06:50:27.769952+0000 front 2024-03-26T06:50:27.770058+0000 (oldest deadline 2024-03-26T06:51:29.370468+0000) 2024-03-26T06:51:38.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:37 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:37.874+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-03-26T06:50:34.372834+0000 front 2024-03-26T06:50:34.372900+0000 (oldest deadline 2024-03-26T06:51:37.872738+0000) 2024-03-26T06:51:39.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:38 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:38.849+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6806 osd.0 since back 2024-03-26T06:48:30.850093+0000 front 2024-03-26T06:48:30.850234+0000 (oldest deadline 2024-03-26T06:50:01.754841+0000) 2024-03-26T06:51:39.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:38 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:38.849+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6804 osd.1 since back 2024-03-26T06:49:02.855790+0000 front 2024-03-26T06:49:02.855714+0000 (oldest deadline 2024-03-26T06:50:34.261605+0000) 2024-03-26T06:51:39.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:38 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:38.849+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6804 osd.2 since back 2024-03-26T06:49:38.962627+0000 front 2024-03-26T06:49:38.962743+0000 (oldest deadline 2024-03-26T06:50:41.862598+0000) 2024-03-26T06:51:39.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:38 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:38.849+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.121:6812 osd.4 since back 2024-03-26T06:49:54.764649+0000 front 2024-03-26T06:49:54.764712+0000 (oldest deadline 2024-03-26T06:51:22.568607+0000) 2024-03-26T06:51:39.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:38 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:38.849+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.170:6812 osd.5 since back 2024-03-26T06:50:27.769952+0000 front 2024-03-26T06:50:27.770058+0000 (oldest deadline 2024-03-26T06:51:29.370468+0000) 2024-03-26T06:51:39.170 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:38 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11[62665]: 2024-03-26T06:51:38.849+0000 7f1f1fbdf700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.38:6822 osd.6 since back 2024-03-26T06:50:34.372834+0000 front 2024-03-26T06:50:34.372900+0000 (oldest deadline 2024-03-26T06:51:37.872738+0000) 2024-03-26T06:51:40.321 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:40 smithi170 podman[75020]: 2024-03-26 06:51:40.008782447 +0000 UTC m=+5.972044130 container died 51a5b8cfc935eb0258df47d352a4f6c1c46eaf84605a55242af9eebab9c7b3c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-26T06:51:40.669 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:40 smithi170 podman[75020]: 2024-03-26 06:51:40.318413239 +0000 UTC m=+6.281674911 container cleanup 51a5b8cfc935eb0258df47d352a4f6c1c46eaf84605a55242af9eebab9c7b3c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T06:51:40.669 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:40 smithi170 bash[75020]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11 2024-03-26T06:51:40.669 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:40 smithi170 podman[75050]: 2024-03-26 06:51:40.494656724 +0000 UTC m=+0.484406731 container remove 51a5b8cfc935eb0258df47d352a4f6c1c46eaf84605a55242af9eebab9c7b3c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:51:41.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:40 smithi170 podman[75139]: 2024-03-26 06:51:40.807131951 +0000 UTC m=+0.017386945 image pull 10cdb6adb6155065d2fa5242628814354d43782bed55cc4f028ed36bb9d0a00c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265 2024-03-26T06:51:41.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:40 smithi170 podman[75139]: 2024-03-26 06:51:40.912172859 +0000 UTC m=+0.122427886 container create 05e1f0bb5790322ec2de29b52230cab746bdaadb4f954f2216c739f3d77533b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T06:51:41.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:41 smithi170 podman[75139]: 2024-03-26 06:51:41.114492035 +0000 UTC m=+0.324747042 container init 05e1f0bb5790322ec2de29b52230cab746bdaadb4f954f2216c739f3d77533b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=rishabh-2024mar8-2-7e98f5f, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T06:51:41.169 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:41 smithi170 podman[75139]: 2024-03-26 06:51:41.117784794 +0000 UTC m=+0.328039779 container start 05e1f0bb5790322ec2de29b52230cab746bdaadb4f954f2216c739f3d77533b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-26T06:51:41.669 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:41 smithi170 podman[75139]: 2024-03-26 06:51:41.197596547 +0000 UTC m=+0.407851546 container attach 05e1f0bb5790322ec2de29b52230cab746bdaadb4f954f2216c739f3d77533b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T06:51:41.669 INFO:journalctl@ceph.osd.11.smithi170.stdout:Mar 26 06:51:41 smithi170 podman[75139]: 2024-03-26 06:51:41.410333672 +0000 UTC m=+0.620588657 container died 05e1f0bb5790322ec2de29b52230cab746bdaadb4f954f2216c739f3d77533b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-osd-11-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-2-7e98f5f, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-26T06:51:41.967 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@osd.11.service' 2024-03-26T06:51:42.007 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:51:42.008 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-26T06:51:42.008 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-26T06:51:42.008 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.a 2024-03-26T06:51:42.334 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:51:42 smithi038 systemd[1]: Stopping Ceph mds.a for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:51:42.674 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:51:42 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-a[71617]: 2024-03-26T06:51:42.333+0000 7f4f95ffb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:51:42.674 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:51:42 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-a[71617]: 2024-03-26T06:51:42.333+0000 7f4f95ffb700 -1 mds.a *** got signal Terminated *** 2024-03-26T06:51:46.613 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:51:46 smithi038 podman[92393]: 2024-03-26 06:51:46.34407532 +0000 UTC m=+4.142013309 container died 7fdcfc8c1d627b605e7e7905a7b381f3e02dc9a5a0f6d442bc8673c4902bcd33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-26T06:51:47.084 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:51:46 smithi038 podman[92393]: 2024-03-26 06:51:46.800270667 +0000 UTC m=+4.598208662 container cleanup 7fdcfc8c1d627b605e7e7905a7b381f3e02dc9a5a0f6d442bc8673c4902bcd33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T06:51:47.085 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 26 06:51:46 smithi038 bash[92393]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-a 2024-03-26T06:51:47.291 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.a.service' 2024-03-26T06:51:47.330 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:51:47.330 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-26T06:51:47.330 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-26T06:51:47.330 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.d 2024-03-26T06:51:47.650 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:51:47 smithi038 systemd[1]: Stopping Ceph mds.d for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:51:48.084 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:51:47 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-d[70659]: 2024-03-26T06:51:47.646+0000 7fcf926d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:51:48.084 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:51:47 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-d[70659]: 2024-03-26T06:51:47.646+0000 7fcf926d3700 -1 mds.d *** got signal Terminated *** 2024-03-26T06:51:51.933 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:51:51 smithi038 podman[92526]: 2024-03-26 06:51:51.654430165 +0000 UTC m=+4.106765807 container died 35bfa5bb4a686081d148620b719e162a90043494fabf6690412a7477cb1557f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-d, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T06:51:52.334 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:51:52 smithi038 podman[92526]: 2024-03-26 06:51:52.029629813 +0000 UTC m=+4.481965446 container cleanup 35bfa5bb4a686081d148620b719e162a90043494fabf6690412a7477cb1557f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T06:51:52.334 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 26 06:51:52 smithi038 bash[92526]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-d 2024-03-26T06:51:52.624 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.d.service' 2024-03-26T06:51:52.660 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:51:52.660 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-26T06:51:52.660 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-26T06:51:52.661 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.g 2024-03-26T06:51:52.941 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:51:52 smithi038 systemd[1]: Stopping Ceph mds.g for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:51:53.334 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:51:52 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-g[71259]: 2024-03-26T06:51:52.975+0000 7f196db47700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:51:53.334 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:51:52 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-g[71259]: 2024-03-26T06:51:52.975+0000 7f196db47700 -1 mds.g *** got signal Terminated *** 2024-03-26T06:51:56.562 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:51:56 smithi038 podman[92646]: 2024-03-26 06:51:56.306704301 +0000 UTC m=+3.432217328 container died 007d8530a19ef5a86c23bfdbdbdf0529662e96071e29bddbd325aad490a05005 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-g, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T06:51:56.830 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:51:56 smithi038 podman[92646]: 2024-03-26 06:51:56.633676967 +0000 UTC m=+3.759189996 container cleanup 007d8530a19ef5a86c23bfdbdbdf0529662e96071e29bddbd325aad490a05005 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T06:51:56.830 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:51:56 smithi038 bash[92646]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-g 2024-03-26T06:51:57.063 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.g.service' 2024-03-26T06:51:57.084 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:51:56 smithi038 podman[92660]: 2024-03-26 06:51:56.827012469 +0000 UTC m=+0.518070627 container remove 007d8530a19ef5a86c23bfdbdbdf0529662e96071e29bddbd325aad490a05005 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-g, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T06:51:57.084 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:51:57 smithi038 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.g.service: Deactivated successfully. 2024-03-26T06:51:57.084 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:51:57 smithi038 systemd[1]: Stopped Ceph mds.g for c2824556-eb39-11ee-95d0-87774f69a715. 2024-03-26T06:51:57.085 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 26 06:51:57 smithi038 systemd[1]: ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.g.service: Consumed 1.079s CPU time. 2024-03-26T06:51:57.097 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:51:57.097 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-26T06:51:57.098 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-26T06:51:57.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.j 2024-03-26T06:51:57.432 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:51:57 smithi038 systemd[1]: Stopping Ceph mds.j for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:51:57.834 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:51:57 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-j[70309]: 2024-03-26T06:51:57.428+0000 7f1bdb018700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:51:57.835 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:51:57 smithi038 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-j[70309]: 2024-03-26T06:51:57.428+0000 7f1bdb018700 -1 mds.j *** got signal Terminated *** 2024-03-26T06:52:01.334 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:52:01 smithi038 podman[92765]: 2024-03-26 06:52:01.019497409 +0000 UTC m=+3.701220191 container died 386c31cfabf7995642a2d4f067f1af14f6856828773921b84a54bf04cc52f7de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-j, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T06:52:01.758 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:52:01 smithi038 podman[92765]: 2024-03-26 06:52:01.496350474 +0000 UTC m=+4.178073259 container cleanup 386c31cfabf7995642a2d4f067f1af14f6856828773921b84a54bf04cc52f7de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-j, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-26T06:52:01.758 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 26 06:52:01 smithi038 bash[92765]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-j 2024-03-26T06:52:01.857 DEBUG:teuthology.orchestra.run.smithi038:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.j.service' 2024-03-26T06:52:01.891 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:52:01.891 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-26T06:52:01.892 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-26T06:52:01.892 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.b 2024-03-26T06:52:02.183 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:52:01 smithi121 systemd[1]: Stopping Ceph mds.b for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:52:02.646 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:52:02 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b[66818]: 2024-03-26T06:52:02.178+0000 7fd274b86700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:52:02.646 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:52:02 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b[66818]: 2024-03-26T06:52:02.178+0000 7fd274b86700 -1 mds.b *** got signal Terminated *** 2024-03-26T06:52:04.732 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:52:04 smithi121 podman[73275]: 2024-03-26 06:52:04.465400727 +0000 UTC m=+2.385726440 container died 6b23d6d3808ec328a9f7b4dff43bc506ec0f8b08a9f36464ccd46428a7900192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T06:52:05.108 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:52:04 smithi121 podman[73275]: 2024-03-26 06:52:04.828897735 +0000 UTC m=+2.749223457 container cleanup 6b23d6d3808ec328a9f7b4dff43bc506ec0f8b08a9f36464ccd46428a7900192 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-26T06:52:05.108 INFO:journalctl@ceph.mds.b.smithi121.stdout:Mar 26 06:52:04 smithi121 bash[73275]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-b 2024-03-26T06:52:05.247 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.b.service' 2024-03-26T06:52:05.285 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:52:05.286 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-26T06:52:05.286 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-26T06:52:05.286 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.e 2024-03-26T06:52:05.646 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:52:05 smithi121 systemd[1]: Stopping Ceph mds.e for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:52:05.646 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:52:05 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-e[66183]: 2024-03-26T06:52:05.583+0000 7f57f17d6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:52:05.647 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:52:05 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-e[66183]: 2024-03-26T06:52:05.583+0000 7f57f17d6700 -1 mds.e *** got signal Terminated *** 2024-03-26T06:52:08.896 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:52:08 smithi121 podman[73410]: 2024-03-26 06:52:08.459474539 +0000 UTC m=+2.965932633 container died 1fd8477c68b1af57e83bd3f2888ec7d1d0ebfbaa47d9e769449aea20c5835f99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-e, ceph=True, org.label-schema.build-date=20240319, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T06:52:09.269 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:52:09 smithi121 podman[73410]: 2024-03-26 06:52:09.073746101 +0000 UTC m=+3.580204186 container cleanup 1fd8477c68b1af57e83bd3f2888ec7d1d0ebfbaa47d9e769449aea20c5835f99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-26T06:52:09.270 INFO:journalctl@ceph.mds.e.smithi121.stdout:Mar 26 06:52:09 smithi121 bash[73410]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-e 2024-03-26T06:52:09.405 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.e.service' 2024-03-26T06:52:09.440 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:52:09.441 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-26T06:52:09.441 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-26T06:52:09.441 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.h 2024-03-26T06:52:09.896 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:52:09 smithi121 systemd[1]: Stopping Ceph mds.h for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:52:09.896 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:52:09 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h[66498]: 2024-03-26T06:52:09.752+0000 7f3ee82ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:52:09.897 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:52:09 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h[66498]: 2024-03-26T06:52:09.752+0000 7f3ee82ea700 -1 mds.h *** got signal Terminated *** 2024-03-26T06:52:12.730 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:52:12 smithi121 podman[73532]: 2024-03-26 06:52:12.455109419 +0000 UTC m=+2.797130556 container died d6d6c8cc8bc586b700a5c1831b912d7988d2d620276141cf9f7cfcb903aae39e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T06:52:13.080 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:52:12 smithi121 podman[73532]: 2024-03-26 06:52:12.810089921 +0000 UTC m=+3.152111057 container cleanup d6d6c8cc8bc586b700a5c1831b912d7988d2d620276141cf9f7cfcb903aae39e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T06:52:13.081 INFO:journalctl@ceph.mds.h.smithi121.stdout:Mar 26 06:52:12 smithi121 bash[73532]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-h 2024-03-26T06:52:13.237 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.h.service' 2024-03-26T06:52:13.271 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:52:13.272 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-26T06:52:13.272 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-26T06:52:13.272 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.k 2024-03-26T06:52:13.646 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:52:13 smithi121 systemd[1]: Stopping Ceph mds.k for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:52:13.646 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:52:13 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-k[65859]: 2024-03-26T06:52:13.555+0000 7f3972801700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:52:13.647 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:52:13 smithi121 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-k[65859]: 2024-03-26T06:52:13.555+0000 7f3972801700 -1 mds.k *** got signal Terminated *** 2024-03-26T06:52:17.849 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:52:17 smithi121 podman[73654]: 2024-03-26 06:52:17.56237338 +0000 UTC m=+4.085110478 container died 15dddf100d4a0c490d13c1ef32881f1fa1b084769d0b0f8fe9d7f64ed291b142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-k, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T06:52:17.850 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:52:17 smithi121 podman[73654]: 2024-03-26 06:52:17.846517584 +0000 UTC m=+4.369254673 container cleanup 15dddf100d4a0c490d13c1ef32881f1fa1b084769d0b0f8fe9d7f64ed291b142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-k, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f) 2024-03-26T06:52:18.146 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:52:17 smithi121 bash[73654]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-k 2024-03-26T06:52:18.146 INFO:journalctl@ceph.mds.k.smithi121.stdout:Mar 26 06:52:18 smithi121 podman[73666]: 2024-03-26 06:52:18.08064634 +0000 UTC m=+0.514068691 container remove 15dddf100d4a0c490d13c1ef32881f1fa1b084769d0b0f8fe9d7f64ed291b142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-k, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T06:52:18.224 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.k.service' 2024-03-26T06:52:18.257 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:52:18.258 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-26T06:52:18.258 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-26T06:52:18.258 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.c 2024-03-26T06:52:18.596 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:52:18 smithi170 systemd[1]: Stopping Ceph mds.c for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:52:18.919 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:52:18 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[71439]: 2024-03-26T06:52:18.592+0000 7f34adb2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:52:18.919 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:52:18 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c[71439]: 2024-03-26T06:52:18.592+0000 7f34adb2d700 -1 mds.c *** got signal Terminated *** 2024-03-26T06:52:20.746 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:52:20 smithi170 podman[75244]: 2024-03-26 06:52:20.456679003 +0000 UTC m=+1.975950329 container died b6177adb81ff81ef4df5d62305cc011235a4e83da6be22ad28c286ee839edb55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:52:21.081 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:52:20 smithi170 podman[75244]: 2024-03-26 06:52:20.851503175 +0000 UTC m=+2.370774474 container cleanup b6177adb81ff81ef4df5d62305cc011235a4e83da6be22ad28c286ee839edb55 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T06:52:21.081 INFO:journalctl@ceph.mds.c.smithi170.stdout:Mar 26 06:52:20 smithi170 bash[75244]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-c 2024-03-26T06:52:21.198 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.c.service' 2024-03-26T06:52:21.236 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:52:21.237 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-26T06:52:21.237 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-26T06:52:21.237 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.f 2024-03-26T06:52:21.419 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:52:21 smithi170 systemd[1]: Stopping Ceph mds.f for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:52:21.918 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:52:21 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f[67728]: 2024-03-26T06:52:21.538+0000 7fdc1d88e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:52:21.919 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:52:21 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f[67728]: 2024-03-26T06:52:21.538+0000 7fdc1d88e700 -1 mds.f *** got signal Terminated *** 2024-03-26T06:52:24.737 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:52:24 smithi170 podman[75376]: 2024-03-26 06:52:24.461099877 +0000 UTC m=+3.014465637 container died 272d2c6aaa37a15aa5ce02dba77c314a574a523d735c5e0b8c1ba4e93f065fe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T06:52:25.109 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:52:24 smithi170 podman[75376]: 2024-03-26 06:52:24.84752923 +0000 UTC m=+3.400894983 container cleanup 272d2c6aaa37a15aa5ce02dba77c314a574a523d735c5e0b8c1ba4e93f065fe7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f, ceph=True, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T06:52:25.110 INFO:journalctl@ceph.mds.f.smithi170.stdout:Mar 26 06:52:24 smithi170 bash[75376]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-f 2024-03-26T06:52:25.264 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.f.service' 2024-03-26T06:52:25.300 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:52:25.301 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-26T06:52:25.301 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-26T06:52:25.301 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.i 2024-03-26T06:52:25.621 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:52:25 smithi170 systemd[1]: Stopping Ceph mds.i for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:52:25.919 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:52:25 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i[68353]: 2024-03-26T06:52:25.617+0000 7f4671e4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:52:25.919 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:52:25 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i[68353]: 2024-03-26T06:52:25.617+0000 7f4671e4d700 -1 mds.i *** got signal Terminated *** 2024-03-26T06:52:28.746 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:52:28 smithi170 podman[75498]: 2024-03-26 06:52:28.456082862 +0000 UTC m=+2.943369212 container died fea61e049736062c4d0d32141be4d3c41746f55d1bf342c5bad55cd8700bf7dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-2-7e98f5f, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T06:52:29.078 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:52:28 smithi170 podman[75498]: 2024-03-26 06:52:28.825649232 +0000 UTC m=+3.312935576 container cleanup fea61e049736062c4d0d32141be4d3c41746f55d1bf342c5bad55cd8700bf7dd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5) 2024-03-26T06:52:29.079 INFO:journalctl@ceph.mds.i.smithi170.stdout:Mar 26 06:52:28 smithi170 bash[75498]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-i 2024-03-26T06:52:29.208 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.i.service' 2024-03-26T06:52:29.244 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:52:29.245 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-26T06:52:29.245 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-26T06:52:29.245 DEBUG:teuthology.orchestra.run.smithi170:> sudo systemctl stop ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.l 2024-03-26T06:52:29.418 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:52:29 smithi170 systemd[1]: Stopping Ceph mds.l for c2824556-eb39-11ee-95d0-87774f69a715... 2024-03-26T06:52:29.919 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:52:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-l[67422]: 2024-03-26T06:52:29.545+0000 7f106d2a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T06:52:29.919 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:52:29 smithi170 ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-l[67422]: 2024-03-26T06:52:29.545+0000 7f106d2a3700 -1 mds.l *** got signal Terminated *** 2024-03-26T06:52:31.169 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:52:30 smithi170 podman[75619]: 2024-03-26 06:52:30.915076214 +0000 UTC m=+1.461173696 container died f2e56bf0d4bd703f3fcccde49ca9d0778ff278d523be5ae4c3daaa4d8b6409a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-l, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-2-7e98f5f, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T06:52:31.441 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:52:31 smithi170 podman[75619]: 2024-03-26 06:52:31.17955104 +0000 UTC m=+1.725648516 container cleanup f2e56bf0d4bd703f3fcccde49ca9d0778ff278d523be5ae4c3daaa4d8b6409a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:cd565c2faf180e3895b642f4837314e69c92d3272b87470dde98807c0b926265, name=ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-l, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-2-7e98f5f, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T06:52:31.442 INFO:journalctl@ceph.mds.l.smithi170.stdout:Mar 26 06:52:31 smithi170 bash[75619]: ceph-c2824556-eb39-11ee-95d0-87774f69a715-mds-l 2024-03-26T06:52:31.603 DEBUG:teuthology.orchestra.run.smithi170:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c2824556-eb39-11ee-95d0-87774f69a715@mds.l.service' 2024-03-26T06:52:31.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T06:52:31.637 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-26T06:52:31.637 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2824556-eb39-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T06:52:31.833 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:52:34.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2824556-eb39-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T06:52:34.506 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:52:37.015 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2824556-eb39-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T06:52:37.208 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:52:39.657 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T06:52:39.692 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T06:52:39.727 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T06:52:39.756 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T06:52:39.758 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752/remote/smithi038/crash 2024-03-26T06:52:39.759 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/crash -- . 2024-03-26T06:52:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:tar: /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T06:52:39.842 INFO:teuthology.orchestra.run.smithi038.stderr:tar: Error is not recoverable: exiting now 2024-03-26T06:52:39.844 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752/remote/smithi121/crash 2024-03-26T06:52:39.845 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/crash -- . 2024-03-26T06:52:39.883 INFO:teuthology.orchestra.run.smithi121.stderr:tar: /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T06:52:39.883 INFO:teuthology.orchestra.run.smithi121.stderr:tar: Error is not recoverable: exiting now 2024-03-26T06:52:39.885 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752/remote/smithi170/crash 2024-03-26T06:52:39.886 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/crash -- . 2024-03-26T06:52:39.933 INFO:teuthology.orchestra.run.smithi170.stderr:tar: /var/lib/ceph/c2824556-eb39-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T06:52:39.933 INFO:teuthology.orchestra.run.smithi170.stderr:tar: Error is not recoverable: exiting now 2024-03-26T06:52:39.934 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T06:52:39.935 DEBUG:teuthology.orchestra.run.smithi038:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-26T06:52:39.969 INFO:tasks.cephadm:Compressing logs... 2024-03-26T06:52:39.969 DEBUG:teuthology.orchestra.run.smithi038:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T06:52:40.012 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T06:52:40.014 DEBUG:teuthology.orchestra.run.smithi170:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T06:52:40.038 INFO:teuthology.orchestra.run.smithi038.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-03-26T06:52:40.038 INFO:teuthology.orchestra.run.smithi038.stderr: /var/log/ceph/cephadm.log 2024-03-26T06:52:40.038 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T06:52:40.038 INFO:teuthology.orchestra.run.smithi038.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T06:52:40.040 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T06:52:40.040 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.6.log: /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.g.log 2024-03-26T06:52:40.040 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.j.log 2024-03-26T06:52:40.040 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T06:52:40.043 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T06:52:40.043 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T06:52:40.043 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mgr.x.log: 90.1% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T06:52:40.047 INFO:teuthology.orchestra.run.smithi121.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T06:52:40.047 INFO:teuthology.orchestra.run.smithi121.stderr:: No such file or directory 2024-03-26T06:52:40.047 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.10.log 2024-03-26T06:52:40.047 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T06:52:40.047 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.b.log 2024-03-26T06:52:40.047 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T06:52:40.047 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T06:52:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.b.log: /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.4.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T06:52:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.k.log 2024-03-26T06:52:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.h.log 2024-03-26T06:52:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T06:52:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.e.log 2024-03-26T06:52:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.7.log: 90.2% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T06:52:40.048 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T06:52:40.052 INFO:teuthology.orchestra.run.smithi038.stderr: 94.9% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.g.log.gz 2024-03-26T06:52:40.052 INFO:teuthology.orchestra.run.smithi038.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T06:52:40.053 INFO:teuthology.orchestra.run.smithi038.stderr: 95.0% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.j.log.gz 2024-03-26T06:52:40.053 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T06:52:40.053 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.log 2024-03-26T06:52:40.053 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T06:52:40.053 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.log: 84.2% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T06:52:40.053 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.9.log 2024-03-26T06:52:40.053 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-volume.log: 86.5% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T06:52:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.e.log: 94.8% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.k.log.gz 2024-03-26T06:52:40.055 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T06:52:40.064 INFO:teuthology.orchestra.run.smithi170.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T06:52:40.064 INFO:teuthology.orchestra.run.smithi170.stderr:: No such file or directory 2024-03-26T06:52:40.064 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.l.log 2024-03-26T06:52:40.064 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T06:52:40.064 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose --/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.l.log: /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.i.log 2024-03-26T06:52:40.065 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.11.log 2024-03-26T06:52:40.065 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mgr.z.log 2024-03-26T06:52:40.065 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.8.log 2024-03-26T06:52:40.065 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T06:52:40.065 INFO:teuthology.orchestra.run.smithi170.stderr: /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T06:52:40.065 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.f.log 2024-03-26T06:52:40.065 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T06:52:40.065 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.f.log: 90.3% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T06:52:40.066 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.log 2024-03-26T06:52:40.066 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T06:52:40.066 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T06:52:40.066 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.log: 94.9% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.l.log.gz 2024-03-26T06:52:40.066 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.c.log 2024-03-26T06:52:40.066 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.2.log: 86.5% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T06:52:40.066 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T06:52:40.066 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.c.log: 91.9% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mgr.z.log.gz 2024-03-26T06:52:40.067 INFO:teuthology.orchestra.run.smithi170.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T06:52:40.067 INFO:teuthology.orchestra.run.smithi038.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.d.log 2024-03-26T06:52:40.068 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.a.log 2024-03-26T06:52:40.068 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.d.log: 90.9% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T06:52:40.068 INFO:teuthology.orchestra.run.smithi038.stderr: 94.9% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.d.log.gz 2024-03-26T06:52:40.069 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mgr.y.log: 91.8% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T06:52:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.log 2024-03-26T06:52:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.cephadm.log: 83.8% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T06:52:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T06:52:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.log: 86.5% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T06:52:40.070 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T06:52:40.070 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-volume.log: 91.0% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T06:52:40.071 INFO:teuthology.orchestra.run.smithi038.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.a.log: 94.8% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.a.log.gz 2024-03-26T06:52:40.078 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mon.b.log: 91.0% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T06:52:40.126 INFO:teuthology.orchestra.run.smithi170.stderr:/var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mon.c.log: 93.3% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.i.log.gz 2024-03-26T06:52:40.133 INFO:teuthology.orchestra.run.smithi121.stderr: 94.5% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.e.log.gz 2024-03-26T06:52:40.142 INFO:teuthology.orchestra.run.smithi170.stderr: 94.6% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.f.log.gz 2024-03-26T06:52:40.306 INFO:teuthology.orchestra.run.smithi121.stderr: 94.2% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.h.log.gz 2024-03-26T06:52:40.363 INFO:teuthology.orchestra.run.smithi038.stderr: 90.0% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T06:52:40.436 INFO:teuthology.orchestra.run.smithi121.stderr: 91.9% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T06:52:40.492 INFO:teuthology.orchestra.run.smithi170.stderr: 91.9% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T06:52:40.824 INFO:teuthology.orchestra.run.smithi038.stderr: 90.4% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T06:52:41.106 INFO:teuthology.orchestra.run.smithi170.stderr: 96.1% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.c.log.gz 2024-03-26T06:52:47.733 INFO:teuthology.orchestra.run.smithi038.stderr: 93.2% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.9.log.gz 2024-03-26T06:52:49.103 INFO:teuthology.orchestra.run.smithi121.stderr: 93.2% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T06:52:51.246 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.8.log.gz 2024-03-26T06:52:51.313 INFO:teuthology.orchestra.run.smithi170.stderr: 93.1% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T06:52:53.441 INFO:teuthology.orchestra.run.smithi121.stderr: 94.8% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-mds.b.log.gz 2024-03-26T06:52:53.853 INFO:teuthology.orchestra.run.smithi038.stderr: 93.1% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T06:52:54.923 INFO:teuthology.orchestra.run.smithi170.stderr: 93.0% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T06:52:54.992 INFO:teuthology.orchestra.run.smithi121.stderr: 93.1% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.10.log.gz 2024-03-26T06:52:55.725 INFO:teuthology.orchestra.run.smithi121.stderr: 93.0% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T06:52:56.548 INFO:teuthology.orchestra.run.smithi170.stderr: 93.0% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.11.log.gz 2024-03-26T06:52:56.550 INFO:teuthology.orchestra.run.smithi170.stderr: 2024-03-26T06:52:56.550 INFO:teuthology.orchestra.run.smithi170.stderr:real 0m16.521s 2024-03-26T06:52:56.550 INFO:teuthology.orchestra.run.smithi170.stderr:user 0m53.558s 2024-03-26T06:52:56.550 INFO:teuthology.orchestra.run.smithi170.stderr:sys 0m1.905s 2024-03-26T06:52:56.788 INFO:teuthology.orchestra.run.smithi038.stderr: 93.0% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T06:52:57.628 INFO:teuthology.orchestra.run.smithi121.stderr: 93.1% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T06:52:57.630 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-03-26T06:52:57.630 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m17.602s 2024-03-26T06:52:57.630 INFO:teuthology.orchestra.run.smithi121.stderr:user 1m9.011s 2024-03-26T06:52:57.630 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m2.271s 2024-03-26T06:52:58.757 INFO:teuthology.orchestra.run.smithi038.stderr: 93.0% -- replaced with /var/log/ceph/c2824556-eb39-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T06:52:58.759 INFO:teuthology.orchestra.run.smithi038.stderr: 2024-03-26T06:52:58.759 INFO:teuthology.orchestra.run.smithi038.stderr:real 0m18.734s 2024-03-26T06:52:58.759 INFO:teuthology.orchestra.run.smithi038.stderr:user 0m56.275s 2024-03-26T06:52:58.759 INFO:teuthology.orchestra.run.smithi038.stderr:sys 0m1.762s 2024-03-26T06:52:58.759 INFO:tasks.cephadm:Archiving logs... 2024-03-26T06:52:58.760 DEBUG:teuthology.misc:Transferring archived files from smithi038:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752/remote/smithi038/log 2024-03-26T06:52:58.761 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T06:53:07.570 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752/remote/smithi121/log 2024-03-26T06:53:07.571 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T06:53:17.258 DEBUG:teuthology.misc:Transferring archived files from smithi170:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752/remote/smithi170/log 2024-03-26T06:53:17.259 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T06:53:25.820 INFO:tasks.cephadm:Removing cluster... 2024-03-26T06:53:25.821 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2824556-eb39-11ee-95d0-87774f69a715 --force 2024-03-26T06:53:26.020 INFO:teuthology.orchestra.run.smithi038.stdout:Deleting cluster with fsid: c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:53:26.351 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2824556-eb39-11ee-95d0-87774f69a715 --force 2024-03-26T06:53:26.545 INFO:teuthology.orchestra.run.smithi121.stdout:Deleting cluster with fsid: c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:53:26.880 DEBUG:teuthology.orchestra.run.smithi170:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c2824556-eb39-11ee-95d0-87774f69a715 --force 2024-03-26T06:53:27.071 INFO:teuthology.orchestra.run.smithi170.stdout:Deleting cluster with fsid: c2824556-eb39-11ee-95d0-87774f69a715 2024-03-26T06:53:27.407 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T06:53:27.407 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T06:53:27.425 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T06:53:27.442 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T06:53:27.457 INFO:tasks.cephadm:Teardown complete 2024-03-26T06:53:27.457 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T06:53:27.468 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T06:53:27.468 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T06:53:27.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T06:53:27.484 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T06:53:27.600 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T06:53:27.611 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T06:53:27.612 DEBUG:teuthology.orchestra.run.smithi038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T06:53:27.615 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T06:53:27.617 DEBUG:teuthology.orchestra.run.smithi170:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T06:53:27.629 INFO:teuthology.orchestra.run.smithi038.stderr:bash: line 1: ntpq: command not found 2024-03-26T06:53:27.629 INFO:teuthology.orchestra.run.smithi121.stderr:bash: line 1: ntpq: command not found 2024-03-26T06:53:27.631 INFO:teuthology.orchestra.run.smithi170.stderr:bash: line 1: ntpq: command not found 2024-03-26T06:53:27.633 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T06:53:27.634 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-03-26T06:53:27.634 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 1 +866us[ +868us] +/- 73ms 2024-03-26T06:53:27.634 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 6 377 0 -1050us[-1049us] +/- 72ms 2024-03-26T06:53:27.634 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 1 +226us[ +227us] +/- 79ms 2024-03-26T06:53:27.634 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:53:27.635 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T06:53:27.636 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2024-03-26T06:53:27.636 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +871us[ +871us] +/- 73ms 2024-03-26T06:53:27.636 INFO:teuthology.orchestra.run.smithi038.stdout:^* hv02.front.sepia.ceph.com 3 6 377 5 -1051us[-1032us] +/- 72ms 2024-03-26T06:53:27.636 INFO:teuthology.orchestra.run.smithi038.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +204us[ +204us] +/- 79ms 2024-03-26T06:53:27.636 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:53:27.637 INFO:teuthology.orchestra.run.smithi170.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T06:53:27.637 INFO:teuthology.orchestra.run.smithi170.stdout:=============================================================================== 2024-03-26T06:53:27.637 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +902us[ +919us] +/- 73ms 2024-03-26T06:53:27.638 INFO:teuthology.orchestra.run.smithi170.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -1013us[ -996us] +/- 72ms 2024-03-26T06:53:27.638 INFO:teuthology.orchestra.run.smithi170.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +227us[ +244us] +/- 79ms 2024-03-26T06:53:27.638 INFO:teuthology.orchestra.run.smithi170.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T06:53:27.638 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T06:53:27.683 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T06:53:27.683 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T06:53:27.717 DEBUG:teuthology.orchestra.run.smithi038:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T06:53:27.804 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T06:53:27.878 DEBUG:teuthology.orchestra.run.smithi170:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T06:53:27.949 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T06:53:27.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:53:27.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T06:53:28.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:53:28.013 DEBUG:teuthology.orchestra.run.smithi170:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-26T06:53:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:53:28.043 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T06:53:28.054 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T06:53:28.125 INFO:teuthology.task.internal:Duration was 2214.397863 seconds 2024-03-26T06:53:28.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T06:53:28.160 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T06:53:28.160 DEBUG:teuthology.orchestra.run.smithi038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T06:53:28.163 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T06:53:28.165 DEBUG:teuthology.orchestra.run.smithi170:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T06:53:28.206 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T06:53:28.207 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T06:53:28.208 INFO:teuthology.orchestra.run.smithi170.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T06:53:28.700 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T06:53:28.700 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi038.front.sepia.ceph.com 2024-03-26T06:53:28.700 DEBUG:teuthology.orchestra.run.smithi038:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T06:53:28.724 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-03-26T06:53:28.725 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T06:53:28.748 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi170.front.sepia.ceph.com 2024-03-26T06:53:28.749 DEBUG:teuthology.orchestra.run.smithi170:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T06:53:28.773 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T06:53:28.774 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T06:53:28.776 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T06:53:28.791 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T06:53:28.843 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T06:53:28.843 DEBUG:teuthology.orchestra.run.smithi038:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T06:53:28.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T06:53:28.860 DEBUG:teuthology.orchestra.run.smithi170:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T06:53:29.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T06:53:29.241 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T06:53:29.241 DEBUG:teuthology.orchestra.run.smithi038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T06:53:29.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T06:53:29.295 DEBUG:teuthology.orchestra.run.smithi170:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T06:53:29.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T06:53:29.334 DEBUG:teuthology.orchestra.run.smithi038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T06:53:29.337 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T06:53:29.339 DEBUG:teuthology.orchestra.run.smithi170:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T06:53:29.359 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = core 2024-03-26T06:53:29.362 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-03-26T06:53:29.387 INFO:teuthology.orchestra.run.smithi170.stdout:kernel.core_pattern = core 2024-03-26T06:53:29.419 DEBUG:teuthology.orchestra.run.smithi038:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T06:53:29.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:53:29.433 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T06:53:29.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:53:29.447 DEBUG:teuthology.orchestra.run.smithi170:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T06:53:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T06:53:29.473 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T06:53:29.486 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T06:53:29.487 DEBUG:teuthology.misc:Transferring archived files from smithi038:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752/remote/smithi038 2024-03-26T06:53:29.487 DEBUG:teuthology.orchestra.run.smithi038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T06:53:29.531 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752/remote/smithi121 2024-03-26T06:53:29.531 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T06:53:29.570 DEBUG:teuthology.misc:Transferring archived files from smithi170:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752/remote/smithi170 2024-03-26T06:53:29.570 DEBUG:teuthology.orchestra.run.smithi170:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T06:53:29.609 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T06:53:29.609 DEBUG:teuthology.orchestra.run.smithi038:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T06:53:29.612 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T06:53:29.614 DEBUG:teuthology.orchestra.run.smithi170:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T06:53:29.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T06:53:29.674 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T06:53:29.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T06:53:29.701 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T06:53:29.702 DEBUG:teuthology.orchestra.run.smithi038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T06:53:29.704 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T06:53:29.707 DEBUG:teuthology.orchestra.run.smithi170:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T06:53:29.717 INFO:teuthology.orchestra.run.smithi038.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 06:53 /home/ubuntu/cephtest 2024-03-26T06:53:29.718 INFO:teuthology.orchestra.run.smithi121.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 06:53 /home/ubuntu/cephtest 2024-03-26T06:53:29.720 INFO:teuthology.orchestra.run.smithi170.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 06:53 /home/ubuntu/cephtest 2024-03-26T06:53:29.722 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T06:53:29.734 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T06:53:29.802 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T06:53:29.832 DEBUG:teuthology.nuke:shortname: smithi038 2024-03-26T06:53:29.833 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T06:53:29.857 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 06:08:11.920262', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDK8IYk/l2NfYVn+XIYVJFZ/6zch67zvao5F9ftyWAOFb4QwIbBKQFDi7jvIcLbmXKXlvVeBZu4ywor6dCJ5w50='} 2024-03-26T06:53:29.862 DEBUG:teuthology.nuke:shortname: smithi121 2024-03-26T06:53:29.862 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T06:53:29.882 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 06:08:11.921872', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJwYVQtiY5yFelKQqCSIngBczcLbFkGF67vYFFUCVmgEC3zPd8pah10Nytdx1k1EFMacI+/uvJgqrL1GXD9xyxo='} 2024-03-26T06:53:29.888 DEBUG:teuthology.nuke:shortname: smithi170 2024-03-26T06:53:29.889 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T06:53:29.913 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi170.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-26_05:47:39-fs-rishabh-2024mar8-2-testing-default-smithi/7622752', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 06:08:11.923122', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH7jjFQL494AaApyg+lSA8QOSmia4/wk6Zer32WH9/pk8u9LEX75LQUUx2pNXlrn17IewuwLjGYpr9f54R6WRa4='} 2024-03-26T06:53:29.920 INFO:teuthology.orchestra.console.smithi038:Power off 2024-03-26T06:53:29.921 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T06:53:29.947 DEBUG:teuthology.orchestra.console.smithi038:power off output: Chassis Power Control: Down/Off 2024-03-26T06:53:29.948 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:53:29.967 INFO:teuthology.orchestra.console.smithi121:Power off 2024-03-26T06:53:29.968 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T06:53:29.981 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-03-26T06:53:29.990 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-03-26T06:53:29.991 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:53:30.010 INFO:teuthology.orchestra.console.smithi170:Power off 2024-03-26T06:53:30.011 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T06:53:30.022 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-03-26T06:53:30.039 DEBUG:teuthology.orchestra.console.smithi170:power off output: Chassis Power Control: Down/Off 2024-03-26T06:53:30.040 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:53:30.065 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-03-26T06:53:33.982 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:53:34.023 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:53:34.067 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:53:34.093 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is on 2024-03-26T06:53:34.134 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-03-26T06:53:34.177 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is on 2024-03-26T06:53:38.098 DEBUG:teuthology.orchestra.console.smithi038:pexpect command: ipmitool -H smithi038.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:53:38.135 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:53:38.178 DEBUG:teuthology.orchestra.console.smithi170:pexpect command: ipmitool -H smithi170.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:53:38.209 DEBUG:teuthology.orchestra.console.smithi038:check power output: Chassis Power is off 2024-03-26T06:53:38.247 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-03-26T06:53:38.289 DEBUG:teuthology.orchestra.console.smithi170:check power output: Chassis Power is off 2024-03-26T06:53:38.310 INFO:teuthology.orchestra.console.smithi038:Power off completed 2024-03-26T06:53:38.390 INFO:teuthology.orchestra.console.smithi170:Power off completed 2024-03-26T06:53:42.250 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T06:53:42.361 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-03-26T06:53:42.462 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-03-26T06:53:42.569 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-bitmap omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/no 3-snaps/yes 4-flush/yes 5-workunit/suites/ffsb}} duration: 2214.3978633880615 failure_reason: 'Command failed (workunit test suites/ffsb.sh) on smithi038 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7e98f5f6b535defc7fa715b456904ffc0500aa9f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh''' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d6753d2be5844c8cba3426e9c8aa116e status: fail success: false 2024-03-26T06:53:42.570 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T06:53:42.658 INFO:teuthology.run:FAIL